guys help im frozen in time

i post more on my FediPub Activityverse: @mothcompute@vixen.zone it is where i talk about all my fun projects


cathoderaydude
@cathoderaydude

if you comment about how windows is bad i will block you. this is not a joke either. do not piss on my effort to help people solve a practical problem they have.

A thing that's always bugged me is that no OS, as far as I know, has a sensible queueing system for concurrent file copies. Like, if you grab a bunch of files in windows explorer, macos finder, or... who am I kidding, nobody uses a gui file manager on linux, but if you did, and then drug them to another window, it'll copy one file at a time, naturally, because duh.

If you then grab more files, from the same or a different folder, and drag them to the same destination, it will start trying to copy both sets of files in parallel, writing blocks semi-randomly from one file or the other instead of finishing one transfer, then performing the second one. This is bad enough on SSDs but if your destination is a spinning disk it'll absolutely destroy your performance, which is what makes it so absolutely wild that this behavior goes back to the 80s, and yeah, I don't think any OS has actually solved for it.

I finally found a way to fix this in Windows though, leveraging a feature that nobody cares about, which they'll probably remove eventually for no reason: Libraries.

Go into Explorer, turn on libraries in the tree view if need be, add a new one, then select all the folders that contain the files you want. Click on the new library, select all the files you want, and copy/cut/drag to the new folder. Windows will now process the whole transfer one file at a time.

I came up with this because I have 6 SSDs, containing 10TB of files, that I need to copy to a spinning disk across a network. I obviously can't sit there and baby the whole process, so I need to start all the transfers at once, but if I pull files from each SSD separately, it'll take 80 hours as it thrashes the disk heads half to death and reduces the transfer speed to kilobytes. The library solution fixes it; I've been getting a clean 250MB/s all day. Good Luck


You must log in to comment.

in reply to @cathoderaydude's post:

does the library method just simplify the process of selecting multiple folders or is there am i missing something special? genuinely asking, this is a neat trick (thanks for posting) but i want to make sure i understand it.

like if i were to use xcopy instead would the result be similar?

also if you are copying tons of small files over the network on regular basis to samba shares i highly using xcopy with the /z (restartable) and /compress (compress over network if dest supports) flags. robocopy is also an option but only really brings multithreading to the table which iirc is only helpful for tons of small files cause samba can choke a bit on those.

this solves for:

A) you don't want every file from each folder. you want a subset of files from several different folders

and/or

B) there is no possible view where you can see all containing folders at once. in this case, I needed to copy files from six different drives. windows, at least, offers no option for "copy the entire contents of each of these entire drives." likewise, if folder A is in the root of the drive and folder B is in a subdirectory, there's no place where you could select both simultaneously.

ok sweet, i got all the intentions!

as to part of point B there's no UI but xcopy is a built in with a bunch of functionality (including cloning entire drives to a destination folder or multiple folders to a single destination with a single command). robocopy is newer and is basically rsync

Ah, thank you! I got this to work, but it was more steps.

  1. If "Libraries" does not appear in your tree view, right-click on the libraries pane (the one with Desktop, Downloads, etc.) and from the context menu, choose Show libraries.
  2. "Libraries" will now appear in the tree view below the libraries pane.
  3. Right-click on "Libraries". From this context menu, choose "Show more options".
  4. A new context menu will appear. Choose "New > Library".

Thank you for this advice! I have several Flash drives to empty and/or fill.

additional aftermarket tool option: teracopy. i'm off windows nowadays but, i bought that. voluntarily. because i liked it. i don't know a better way to praise a random piece of software.

it can do all kinds of fancy stuff but it can also just replace the explorer move/copy function with one that does the right thing

edit: i'm a minute late. oh well. let this stand as another recommendation for it

Honestly, I'm pleasantly surprised that there is a good gui way to do this at all, and props to whatever presumably minnow-sized team within the sprawling ocean of windows os devs at ms created such a thing. Is it related at all to the "my briefcase" sync folder that 95/98/2k had?

fwiw, thunar on linux does queue subsequent copies like this. (which has actually annoyed me massively! if i fail to plan correctly, and start a copy of an enormous batch of files but then realise i'd really like one small file copied Right Now, i cannot do that from the gui without having to press either the "pause" or "stop" button and gods only know what either of those will do)

Teracopy has this has a built-in feature which is why I've kept it as an always-install tool in Windows even as their default filecopy program has gotten generally better -- but the fact the default still lacks basic queueing is really bizarre. I shouldn't have to install a third-party app for this lmao