You really think I am going to go out and buy a new and different OS system to get faster file transfers?
Then you are talking about the file system, and there's no file system that I know of that would help the small file transfer speed, and it's not that dependent on file 'overhead' such as allocation unit size.

I think you missed what I said.

When you go out and buy something new, you have at that time opportunity to read and understand the specs of that device, you can reserach what "OS" or chipset it runs, and then buy the better item if costs are close or equal.

I never said you go out and buy replacements, but many people do.


Allocation is not huge in overhead, it's relatively small amount of time to fill allocation after file contents. The overhead is all the CPU cycles and I/O in extra OS record keeping of the file being created. File create is more overhead than just re-writing into existing file. Devices that lack a full OS still need to do recording keeping of a file, otherwise it would have no way to access such file.

And yes, filesystem type makes a difference. Try file transfer FAT to FAT, FAT to NTFS, NTFS to NTFS. On linux do the same EXT4 to EXT4, EXT4 to F2FS, F2FS to F2FS. F2FS is about 8% faster than EXT4.

So, filesystem is very important when you frame stuff in "speed" context. Being able to tune parameters of the filesystem is also important, many times you can't because the device is hard coded and you have no way to change a parameter.

 

I am not sure it is worth talking about this too much though because we can only change so much with our own systems and we are sort of stuck with whatever technology we get thrown at us.

I think that's a good summary of it. It also appears to be what we deal with all the time, evolution of technology in the marketplace. But in many cases you have a choice, I should perhaps upgrade some of my older USB stuff for newer better USB3.2 or even USB4, just so I can save 10sec, or maybe 2min, when doing file transfers.

It would perhaps be much more beneficial to end-users is the device could be upgraded to support newer protocol stuff, but I guess makers of the throw-away stuff don't care to sell just one thingy.

 

I think you missed what I said.

When you go out and buy something new, you have at that time opportunity to read and understand the specs of that device, you can reserach what "OS" or chipset it runs, and then buy the better item if costs are close or equal.

I never said you go out and buy replacements, but many people do.


Allocation is not huge in overhead, it's relatively small amount of time to fill allocation after file contents. The overhead is all the CPU cycles and I/O in extra OS record keeping of the file being created. File create is more overhead than just re-writing into existing file. Devices that lack a full OS still need to do recording keeping of a file, otherwise it would have no way to access such file.

And yes, filesystem type makes a difference. Try file transfer FAT to FAT, FAT to NTFS, NTFS to NTFS. On linux do the same EXT4 to EXT4, EXT4 to F2FS, F2FS to F2FS. F2FS is about 8% faster than EXT4.

So, filesystem is very important when you frame stuff in "speed" context. Being able to tune parameters of the filesystem is also important, many times you can't because the device is hard coded and you have no way to change a parameter.

Hi,

I am not sure but it sounds like you are basing your file system argument on something around 8 percent in speed increase. For me, if it was only 8 percent I'd be perfectly happy. I was seeing 80 percent or even 800 percent differences, which never made sense to me. How could 10000 files of 100k take so much longer than a single 1GB file to copy to another hard drive.
If a 1GB file took 5 seconds and the 10000 files took 120 seconds, so one is 200MB/s and the other is 8MB/s, and that's just 4 percent of the transfer speed of the 1GB file. That's so different it makes no sense to me no matter what facts I look at. With the allocation unit being 4096 bytes, that means the max disk space wasted would be less than 5 percent. But even if the waste was 100 percent (200k for a 100k file) the transfer speed would only be 16MB/s (considering all the bytes rather than just the 100k file data bytes) which is very strange.

This led me to think that there was something up with the SATA protocol. We went from 16 data lines with EIDE down to just 2 lines with SATA. I'm not sure that's the cause anymore though I think it may have something to do with the way it has to handle multiple requests.
Oh almost forgot, disk tests show this behavior also. 500MB/s transfers for larger files, if we get lucky we get 10MB/s transfers for a lot of small files. Back in the day, 10MB/s was even slow for programmed I/O operations with the more typical 20MB/s or even 30MB/s.

I never found a good explanation for this.

Anyway, I am fairly happy with the USB 3.0 and USB 3.2 gen 2 specs. USB 3.0 works ok I guess and USB 3.2 gen 2 is what I am always after now. That combined with the external SSD M2 form disks works pretty nice, even though I think the transfer of smaller files should be faster with those too.

 

I am not sure but it sounds like you are basing your file system argument on something around 8 percent in speed increase.

I don't know everything in my head, I looked up some info on various filesystems. F2FS shows as 8% faster than EXT4. Duly noted, filesystems have "features" and some have tunable parameters. F2FS may be faster, but EXT4 may be "better". Depends on the needs.

USB has nothing to do with filesystem as we know, I say run USB4 if you can to get the fastest data rate on USB.