Server recommendation for *very heavy* CPU/disk task
Hi - I need a bit of help with a recommendation for a server.
I have a huge amount of data in tiny JSON files that I need to load into a MySQL database. There could be nearly 100 million files that I need to parse, each about 500-1000 bytes in size.
At the moment, I need to parse through the files sequentially, so it takes a long time, but in the months ahead I should be able to handle them in parallel and split them into chunks, and then I'll be able to fully utilise multiple cores on a server. I am doing a fair bit of caching with Redis to speed things up, but this is still a huge job overall. Currently using a CPX21 with Hetzner, unfortunately only one core really being used for the actual 'parse' process.
I'm not sure what would be best for this - a high spec Ryzen VPS with maybe 4-6 cores, 4GB+ RAM, and as fast as possible NVMe disk, of at least 100GB NVMe storage space -- or should I be considering a dedicated server, so I know that the I'll be getting full use of the resources all the time? VPS prices are more attractive, but would I have full speed of the NVMe disk all the time?
I'm trying to keep costs down, as this is a bit of a hobby project at the moment - but I think being Black Friday, now is a good time to get a hefty server that can handle the job better than my current 'load box'. Ideally I would want to snag a bargain in the days ahead and get a really high spec server for EUR 100 or less, that will do the job well for me over the year ahead.
Thoughts? Thanks in advance.