Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Scrapers running on a VPS
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Scrapers running on a VPS

pikzpikz Member
edited November 2011 in General

Are there any potential problems running scrapers on a VPS? I assume some providers do not allow it, but even if they would allow it, what should be taken into consideration before running a scraper on a VPS? Could it for instance take up too much CPU power that it would not be considered appropriate? At which point have you crossed the line?

Thanks for any feedback about this.

Comments

  • I would believe that you can run that with almost any VPS, as long as it doesn't use too much bandwidth...

    If you buy a VPS but your provider won't let do anything with it, better spend a few bucks more and go with another provider imho!

    Thanked by 1pikz
  • It's not so much the CPU power part as the warez/copyright infringement/theft part.

    Thanked by 1pikz
  • drmikedrmike Member
    edited November 2011

    The lesser intelligent folks run the scraper and the site that hosts the scraped data on the same IP address. Moving the data elsewhere would be a huge plus and allow you to get away with it. Takes 2 seconds to match those up and file a complaint with the host.

    Or even on a VPS with 2 IP addresses on different class c ranges, one for the scraper and one for the hosted site.

    We don't allow such to happen on our sites but some providers allow things like spam to happen if the site is on their hardware but the spam is actually coming from outside their network.

    edit: I was going to call them 'stupid' folks but well....

    Thanked by 1pikz
  • As long as you are only scraping (not posting/commenting/spamming) then you shouldn't run into any complaints, as most sites will already have their own protection against scrapers (displaying captchas)... Only exception would be if you're scraping so much that it looks like a ddos.

    In regards to CPU, it depends on the scraper... I have custom ones that only use <1% CPU but then I have others that sit around 5-10% CPU. You'll need to test your script out before hand.

    Along with what drmike said, if you're going to host a site on the same VPS, make sure to get another IP.

    (It's also smart to get a handful of proxies and tunnel all traffic through them so you will never have to worry about your VPS getting complaints).

    Thanked by 1pikz
Sign In or Register to comment.