Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Sorting 76k files?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Sorting 76k files?

J1021J1021 Member

I have a folder with 76k files in it, all small image files. They are all images named 033000_070614.jpg etc, the first half of the name being the time it is generate and the remaining half the date. I need all of these files sorting into folders by date and am currently dragging and dropping them via SFTP using FileZilla but it taking an age. The server is really taking it's time to move the files, I assume because of it having 76k inodes in one directory.

Is there any sort of command I could use to move them all into the appropriate folders? Am using Debian.

Comments

  • Drag and drop 76,000 files? That's harsh.

    If nobody comes up with something for you before. I'll make a script for you :)

  • J1021J1021 Member

    ElChile said: Drag and drop 76,000 files? That's harsh.

    If nobody comes up with something for you before. I'll make a script for you :)

    It's not a case of dragging and dropping 76,000 individual files. The server they're on has a GUI via VNC. Loading them up via the file explorer really lags the server. So at the moment I am logging in locally from the server to the server using FSTP/FileZilla and dragging and dropping around 3k at a time. It takes about 45-60 minutes for each 3k files this way but atleast I can do it and leave it running on the server.

    Thanked by 1ElChile
  • Put them on a .tar file and then download the .tar

    Thanked by 1Gunter
  • J1021J1021 Member

    albertdb said: Put them on a .tar file and then download the .tar

    Then I've got to upload them back to the server over my 1Mb/s ADSL connection.

  • BlazeMuisBlazeMuis Member
    edited June 2014

    Try this:

    find /path_of_files -iname "*.jpg" -type f -exec mv {} /path_to_destination_folder \;
  • @1e10 said:
    Then I've got to upload them back to the server over my 1Mb/s ADSL connection.

    What do you want to do?

    Sorting them is the goal or is way to solve the lag?

  • J1021J1021 Member

    albertdb said: Sorting them is the goal or is way to solve the lag?

    Sort them for storage on the server.

  • I'm sorry to say this, but, who the heck would put 100 000 image files in a folder on a server? Especially when they're on a 1Mbps ADSL at home.

  • J1021J1021 Member

    kerouac said: I'm sorry to say this, but, who the heck would put 100 000 image files in a folder on a server? Especially when they're on a 1Mbps ADSL at home.

    There is a script on the server that generates an image every 30 seconds. They're stored in a folder where I then sort them into dates. I got a bit behind sorting them into their folders and now there are 76k files in one folder.

  • albertdbalbertdb Member
    edited June 2014
    for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14
    do
    for j in 01 02 03 04 05 06 07 08 09 10 11 12
    do
    mkdir -p $i/$j
    mv *_$i$j* $i/$j/
    done
    done
    
  • J1021J1021 Member

    Might spin up a Vultr instance, download them all and then upload them all.

  • r0t3nr0t3n Member
    edited June 2014

    #!/bin/bash for i in $(ls *_*.jpg); do d=$(echo $i | cut -d '_' -f 2 | cut -d '.' -f 1) if [ ! -d "$d" ]; then mkdir $d echo "created $d" fi mv $i $d echo "$i moved to $d"; done

    If you don't want the commentary, comment out the echo's

  • J1021J1021 Member

    Thanks for the all help guys. The Vultr instance seems to be managing the task fine.

  • c0yc0y Member

    1e10 said: Thanks for the all help guys. The Vultr instance seems to be managing the task fine.

    That's pretty harsh, to ask for advice and then just ignore everyone and persuade your own ingenious thoughts

  • J1021J1021 Member

    c0y said: That's pretty harsh, to ask for advice and then just ignore everyone and persuade your own ingenious thoughts

    Wasn't the intention.. touchy

  • c0yc0y Member

    1e10 said: Wasn't the intention.. touchy

    You are free to judge me, but just to confirm we're on one line:

    1e10 said: Is there any sort of command I could use to move them all into the appropriate folders? Am using Debian.

    Did you run any of the given scripts or just spun up that Vultr instance and did what your other VPS couldn't?

    Not trying to pick on you here (= this post), this is merely to conform my curiosity.

  • J1021J1021 Member

    c0y said: Did you run any of the given scripts or just spun up that Vultr instance and did what your other VPS couldn't?

    I don't believe that is a condition or string of posting here. I've thanked everyone for their input already, so leave it.

  • c0yc0y Member

    @1e10 said:
    I don't believe that is a condition or string of posting here. I've thanked everyone for their input already, so leave it.

    You call me touchy, yet you won't even answer my curiosity. I won't have expected anything else from a person like you, calling other people what you are yourself.

  • J1021J1021 Member

    c0y said: You call me touchy, yet you won't even answer my curiosity. I won't have expected anything else from a person like you, calling other people what you are yourself.

    What is there to answer? I asked for some solutions and found one of my own, big deal.

  • c0yc0y Member

    1e10 said: What is there to answer?

    c0y said: Did you run any of the given scripts or just spun up that Vultr instance and did what your other VPS couldn't?

    ^This, because that is what I was curious about.

    1e10 said: I asked for some solutions and found one of my own, big deal.

    But don't bother answering it, my curiosity isn't worth trying to break your rebellious stance, because apparently it is a big deal to you.

    I'm out.

    Thanked by 1ErawanArifNugroho
  • J1021J1021 Member
    edited June 2014

    Wow. Mods, lock this thread please?

  • c0y said: But don't bother answering it, my curiosity isn't worth trying to break your rebellious stance, because apparently it is a big deal to you.

    What a douche.

  • ReeRee Member

    @1e10 said:
    There is a script on the server that generates an image every 30 seconds. They're stored in a folder where I then sort them into dates. I got a bit behind sorting them into their folders and now there are 76k files in one folder.

    I see you already solved the backlog, but going forward wouldn't it make sense to modify the script so that you don't need to do this manual sorting anymore? Is that just not possible for some reason?

  • Wasn't the intention

    That's fine, glad you found an answer... and maybe you learned something. The bash scripts people made the effort to post look like they'd do the trick and automate it for the future.

  • douchebag

  • vedranvedran Veteran

    Problem solved and the thread is going in a really strange direction, so closed.

    Thanked by 1Mark_R
This discussion has been closed.