Does FAT Suck?

[Мастер]

I have a solid-state USB drive I use to move files from one place to another. One folder has about 28,000 files, taking up about 2GB. If I try to copy it to the USB, it will take all day. If I try to copy to a larger USB, with a traditional magnetic-medium hard drive inside, it takes maybe five minutes. The larger disk is formatted NTFS.

[Lance]

The write speed to a disk is much faster than to a flash drive.

[Мастер]

Hmm, I had suspected the solid-state would be faster than the physical medium, but that was without doing any research on the matter.

The difference is extreme, though - a couple of minutes vs. all day, or several days. And both of them are getting squeezed through the USB interface . . .

[Lance]

You can get higher speed flash memory but you'll pay a preium price for it. Look at memory cards for digital cameras for an idea of the price differences. But yeah, flash drives are slow.

Though... I don't know if FAT plays any role here or not. Why not reformat the flash drive as NTFS and see if it makes a difference?

[Мастер]

Though... I don't know if FAT plays any role here or not. Why not reformat the flash drive as NTFS and see if it makes a difference?

I was planning to do exactly that!

[Мастер]

Well, I ended up doing a somewhat different experiment.

Copied a folder, containing > 140 sub-folders, and > 28,000 files (mostly pretty small), from the internal NTFS-formatted hard drive to the FAT-formatted USB flash drive. Both experiments conducted on the home Dell desktop, a few years old.

Booted up in a home-brew Linux - copying the folder and the contents took just under 7.5 minutes. Compared the contents, using the "diff" command, didn't time it, but it seemed pretty quick. (Probably less than one minute.)

Booted up in Windows XP - just deleting the contents is taking many, many times as long as copying them there under Linux did. More than 30 minutes, and according to the progress bar, still only a small fraction of the job is completed.

Haven't run a controlled experiment on the HP laptop craputer, running Windows 7, but anecdotally have found copying to/from the flash drive to be painfully slow.

My speculative conclusion - it is not FAT per se that sucks, but the Windows driver for FAT. I suspect it is not buffering anything in memory - each of the little files that get copied, the entire block of memory gets loaded from the flash drive into RAM, is then modified in RAM, and then written back to the flash drive, even if the file only takes up a small piece of one block of memory. Whereas Linux does buffering, keeping recently used flash memory pages in RAM, so the block doesn't actually get written back to the flash drive until after it is completely modified, probably with many of the little files.

At least, that's my best guess.

[Мастер]

Nice. After chugging away for more than an hour, the Windows progress bar now has a time estimate - only 240 minutes remaining, to delete the files from the USB, which were copied there in 7.5 minutes under Linux, running on the same hardware :(

[Lance]

Hmmm, you can change the cache settings.

1) Stick it in
2) Find it under Disk Drives in Device Manager
3) Right Click -> Properties
4) Policies

You can set it for Quick Removal or Better Performance

Give that a shot...

But I also think NTFS will be quicker too.

[Мастер]

My first inclination was to respond, "Holy crap Batman, you mean I've been suffering needlessly all this time?" However, I actually tried it first, and . . .

It was set to optimise for quick removal, but I can't tell that optimising for performance has really changed that much :( Quite possibly faster than it was, but still many times slower than the Linux copy.

[Lance]

Rats!

Have you tried formatting it NTFS yet?

[Lance]

Okay, I just did some transfer testing.

This is NOT what I was expecting to see. Is your flash drive FAT or FAT32?

Seconds to transfer 1 70MB file to a 128MB Flash Drive:

FAT: 15
FAT32: 15
NTFS: 15

Seconds to transfer 117 files totaling 70MB to a 128MB Flash Drive:

FAT: 15
FAT32: 110
NTFS: 33

ETA:

And here is an interesting article:

http://www.irongeek.com/i.php?page=secu ... filesystem

[Enzo]

I really REALLY know nothing of all this, but that doesn't keep my mouth closed very often...

Is deleting a specific process? In other words could it be faster to MOVE the files to someplace than delete them? The kill them wherever they went?

Not a well formed thought, just a notion.

[Мастер]

I really REALLY know nothing of all this, but that doesn't keep my mouth closed very often...

Is deleting a specific process? In other words could it be faster to MOVE the files to someplace than delete them? The kill them wherever they went?

Not a well formed thought, just a notion.

My guess is that the move would take horrifically long as well, but it's just a guess - I haven't performed that particular experiment. (And can't, until I have the USB repopulated - currently empty.)

Lance - when I click on the "properties" (Windows XP), it says this is "FAT" - that means FAT16, right?

One thing I have noticed - when I did the right click the folder on this thing, and chose "properties", the list came up almost instantly. On the bigger, magnetic USB drive, it took a while. Maybe that has to do with caching.

[Lance]

Lance - when I click on the "properties" (Windows XP), it says this is "FAT" - that means FAT16, right?

Yessir.

[Мастер]

Formatted NTFS now. This involved logging out, and back in as an administrator, which would have meant abandoning my post at IRU, which I didn't want to do. (I think later versions of Windows are better at this.) Choices when formatting were "FAT", "FAT32", "NTFS", and "exFAT".

The copy is still in progress, and we are definitely going to come in much slower than it was under Linux with FAT, but nothing like the horror show it was with Windows XP and FAT. If the progress bar is accurate, we'll come in under one hour - it was more like days with FAT.

This is Windows XP - similar behaviour with the laptop running Windows 7, but that laptop has so many serious issues, I'm not sure we should really draw any conclusion from that . . .