New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Sorting 76k files?
I have a folder with 76k files in it, all small image files. They are all images named 033000_070614.jpg etc, the first half of the name being the time it is generate and the remaining half the date. I need all of these files sorting into folders by date and am currently dragging and dropping them via SFTP using FileZilla but it taking an age. The server is really taking it's time to move the files, I assume because of it having 76k inodes in one directory.
Is there any sort of command I could use to move them all into the appropriate folders? Am using Debian.
This discussion has been closed.
Comments
Drag and drop 76,000 files? That's harsh.
If nobody comes up with something for you before. I'll make a script for you
If nobody comes up with something for you before. I'll make a script for you
It's not a case of dragging and dropping 76,000 individual files. The server they're on has a GUI via VNC. Loading them up via the file explorer really lags the server. So at the moment I am logging in locally from the server to the server using FSTP/FileZilla and dragging and dropping around 3k at a time. It takes about 45-60 minutes for each 3k files this way but atleast I can do it and leave it running on the server.
Put them on a .tar file and then download the .tar
Then I've got to upload them back to the server over my 1Mb/s ADSL connection.
Try this:
What do you want to do?
Sorting them is the goal or is way to solve the lag?
Sort them for storage on the server.
I'm sorry to say this, but, who the heck would put 100 000 image files in a folder on a server? Especially when they're on a 1Mbps ADSL at home.
There is a script on the server that generates an image every 30 seconds. They're stored in a folder where I then sort them into dates. I got a bit behind sorting them into their folders and now there are 76k files in one folder.
Might spin up a Vultr instance, download them all and then upload them all.
#!/bin/bash for i in $(ls *_*.jpg); do d=$(echo $i | cut -d '_' -f 2 | cut -d '.' -f 1) if [ ! -d "$d" ]; then mkdir $d echo "created $d" fi mv $i $d echo "$i moved to $d"; done
If you don't want the commentary, comment out the echo's
Thanks for the all help guys. The Vultr instance seems to be managing the task fine.
That's pretty harsh, to ask for advice and then just ignore everyone and persuade your own ingenious thoughts
Wasn't the intention.. touchy
You are free to judge me, but just to confirm we're on one line:
Did you run any of the given scripts or just spun up that Vultr instance and did what your other VPS couldn't?
Not trying to pick on you here (= this post), this is merely to conform my curiosity.
I don't believe that is a condition or string of posting here. I've thanked everyone for their input already, so leave it.
You call me touchy, yet you won't even answer my curiosity. I won't have expected anything else from a person like you, calling other people what you are yourself.
What is there to answer? I asked for some solutions and found one of my own, big deal.
^This, because that is what I was curious about.
But don't bother answering it, my curiosity isn't worth trying to break your rebellious stance, because apparently it is a big deal to you.
I'm out.
Wow. Mods, lock this thread please?
What a douche.
I see you already solved the backlog, but going forward wouldn't it make sense to modify the script so that you don't need to do this manual sorting anymore? Is that just not possible for some reason?
That's fine, glad you found an answer... and maybe you learned something. The bash scripts people made the effort to post look like they'd do the trick and automate it for the future.
douchebag
Problem solved and the thread is going in a really strange direction, so closed.