New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Python Script runs slower compared to my PC?
On my PC it runs almost 10x faster compared to my rented Server?
Any ideas on what the issue could be?
Server Specs:
RAM: 4 GB DDR3
Disk: 15 GB SSD
CPU: 3 Cores @ 3.30 GHz - (Intel(R) Xeon(R) CPU E5-2643 0 @ 3.30GHz)
Bandwidth: 5 TB @ 1 GBps
My PC:
RAM: 32 GB DDR4
Disk: 1 TB HDD + 256 GB SSD
CPU: 4 Cores @ Intel Core i7-6700HQ @ 2.59 GHz
Bandwidth: Unlimited @ 1 GBps
Comments
Just by looking at how misbalanced the specs are - 15GB SSD for 4GB RAM and 3x Intel Xeon Cores, I can already have a guess the node is terribly oversold on CPU and RAM.
Looks normal to me.
Well for one you have a newer CPU on your PC compared to your server. Also you don't have the full CPU on the server either, you are sharing it with other people so it will be slower.
Any provider that can run my Python Script and provides dedicated resources? I'll need to priorities bandwidth mostly and CPU. I'll be making an external connection to my database.
Any suggestions? I need it to run at the same speed as my PC since this server takes way too long.
Thank you!
Sure, go for a dedicated server.
A VPS will always be shared no matter what they claim.
Don't want a dedicated server for like $40/mo if all I am going to do on it is run a simple Python Script that loops. Also need something in the EU since that's where my other server is.
You get what you pay for.
Then my advice is: get used to slower speed.
You aren't going to get a cheap powerful dedi. You can either try a VM from @seriesn which is based off a Ryzen CPU which has a higher clock speed or continue to run it on your PC at home.
Well, i7 has likely higher single core perf + clockspeed, does not suprise me.
Yep and his VMs cpu is 8 years old too.
Pythons are generally slower.. Try Cobra or Viper
Not sure what your use case is but it was a no-brainer for me to channel the amount for a dedicated into my own desktop to do big data and machine learning work. Whatever I would have paid to rent a powerful dedicated for a year more than paid my machine and I get to use it for easily a couple more years. If you need an always on machine running a script, I wonder why you need so much power unless you are doing machine learning of some sort.
I just want it to run as smooth as it runs on my PC or close enough at least.
It's probably because it's constantly making a connection to my website, reading data, downloading data and so on and the server does not seem to like it.
I need it running 24/7 since it is a script that is connected to my 2nd site and it updates data for me.
data updating does not need to be that smooth, unless you need real time updates?
Smooth as in it's taking almost 30 minutes or so to run through my query whereas on my PC it takes < 10 minutes.
Python Script Monitoring (according to my PC)
It is called every 15 minutes, and it is completed in less than 5 minutes on my PC.
So I guess the server I have is just not good enough to handle my script since the specifications are outdated maybe?
Your comparing apples to oranges. You are comparing using all of the cores fully on your local desktop system, compared to using 3 shared cores on another server that isn't even the same CPU. This is why people have already told you, if you want the benefit of being able to use all of the CPU resources for your self, then you need to get a dedicated server instead of a VM where you are sharing your resources with other users on the server.
Would be like saying, I have a v6 Mustang and it can go 0-60 in 3 seconds and I also have a v4 mustang and it only does 0-60 in 6 seconds, why is the one with the smaller engine going slower?
my 2 cents.
Cheers!
If by "query" you mean a DB query, you're doing something wrong.
Perhaps if you tell us a little more about what you're doing we could suggest some improvements.
Running python on a 3rd box to update box #2 from box #1 already seems suboptimal.
[Channeling inner Jsg]
Don't you tell me what to do, you ain't my dad!
Well it's not exactly doing that. But that's the usage it is taking up. Maybe the server has an issue? If its taking 30 minutes to complete and my PC does it in less than 5?
I thought I had posted the server usage as well but oh well. I do agree that I am using shared resources but it shouldn't struggle this much at most it should complete it in 10-15 minutes and not 30. I think I'll try out a new provider with similar specifications to see if the issue persists.
Join the family
Tried on a,
Surprisingly much faster than my PC, completed my results in less than 20 seconds. I think I've found my server choice for now since it has Unmetered and I'm not sure how much I'll be using since it runs every 15 minutes per day. Thank you for the message though @seriesn.
@sgno1 Can you share what the script is doing? You should be able to profile it, and find the areas that are slow.
Is it using a remote database? Is there latency present when you run on a VPS that isnt present on your laptop? 100,000 records with 100ms of latency each adds up.
Is it using alot of disk IO? NVME is going to be significantly faster for reading/writing small files. You can usually find ways to parallelize out of this, but single threaded python that is using the disk will benefit immensly from NVME.
Is it CPU bound? I doubt it -- both processors are fairly comperable unless you are seeing CPU steal.
User PID CPU% MEM%
root 20951 4.0 1.3
That is what its used since I've let it run last night. The script is reading data from a website, downloading it (if it doesn't exist) & storing into a database and it repeats this process every 15 minutes. It works with it's own local database. And I don't think there is a latency since the website it is downloading data from/reading from seems to also be located in the EU so it should be fairly fast.
@sgno1 We do quite a bit of Python based consulting. I'd be happy to take a quick look with you and determine whats generating the latency -- though it appears you found a host that is more performant. The offer stands if you want to review it in the future, happy to sign an NDA.