Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Advertise on LowEndTalk.com
Where is a good location to host a Usenet Indexer?
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Where is a good location to host a Usenet Indexer?

yowmamasitayowmamasita Member
edited January 2013 in General

I'm working on a usenet indexer project Ive been building from scratch. Where's a good location to host it? Thanks!

I'm not downloading copyrighted material, I just grab headers from it. Nzbmatrix, a usenet index site, reportedly received massiv DMCA takedowns and decided to close.

dog lover

Comments

  • Since you're not downloading copyrighted material, we can host you. We have a few other people running nzb*-esque programs.

  • trewqtrewq Administrator, Moderator, Provider

    @Damian What would you do if DMCA take down notices were issued?

  • jarjar Provider

    @trewq said: What would you do if DMCA take down notices were issued?

    Every now & then you gotta have some balls and tell them to shove it. Not a step you want to take on faith, but when you know you're on solid ground, don't let them bully you.

    MagicSpam blackmails providers into buying their software, and ServerHub is a professional spam organization.

  • @trewq said: What would you do if DMCA take down notices were issued?

    If they're valid, we're required to act upon them. So far, out of ~50 DMCA notices received in our two years of operation, only two have been valid so far.

  • @Damian said: If they're valid, we're required to act upon them.

    How do you know if it's valid? Sorry if this is a dumb question but I really have no idea

    dog lover

  • @yowmamasita said: How do you know if it's valid? Sorry if this is a dumb question but I really have no idea

    Majority (if not all) of the DCMAs are all automated by bots. They pick up a TON of false positives.

    Catalyst Host - Pie Approved!
  • WilliamWilliam Member, Provider

    I thought about running one, but the legal implications in UK (where my company is), Liechtenstein (where the holding is) and my location (Slovenia) told me it's not good to do it... beware of the attached risks, don't be naive.

    AFAIK it should be fine in the US however, if you react to valid DMCA.

  • NekkiNekki Moderator

    I don't think indexing usenet headers is actually illegal anywhere, is it?

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • @Nekki said: I don't think indexing usenet headers is actually illegal anywhere, is it?

    That's what i wonder too. But why is it that a number of usenet indexers are shutting down?

    dog lover

  • NekkiNekki Moderator

    Have any pure indexers been shut down? I thought it was just the sites that took submissions from users so quality was higher, as opposed to raw indexers like binsearch.info

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • They shut down because their payment gateways don't cooperate.

  • stupid question, what is the usage of usenet indexer?

    ...............

  • NekkiNekki Moderator

    @kampung said: stupid question, what is the usage of usenet indexer?

    So rather than having to trawl through all the various usenet groups separately to find what you want, you have a consolidated index of all the post headers (usually searchable) to help you track down things. It's basically a search engine for usenet, but most sites include a few more bells and whistles.

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • prae5prae5 Member
    edited January 2013

    http://www.lowendtalk.com/discussion/6996/newznab-friendly-host

    I've run it on a few different providers and all have been accepting of it once confirmed it was an indexer only and no content was being downloaded.

    Currently have http://get.nzb.es running with http://getkvm.com

  • NekkiNekki Moderator

    @prae5 said: I've run it on a few different providers and all have been accepting of it once confirmed it was an indexer only and no content was being downloaded.

    Currently have http://get.nzb.es running with http://getkvm.com

    Out of curiousity, what spec are you running that on, and how much are you indexing?

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • Currently running on a GetKVM node:

    RAM: 2560
    SWAP: 2560
    DISK: 45GB
    TRAFFIC: 45GB
    PORT: 100Mb

    I've run it on a machine with 512Mb in the past and performance was fairly acceptable.

    From the testing I've been doing, RAM helps a little but the biggest influence effecting the server load is disk IO.

    Before running it on this node I tested on a 512Mb node that was SSD based and load was pretty much nothing. I moved it to a NodeDeploy server with 2GB RAM/1GB SWAP and again load was seldom above .2 when indexing and backfilling groups.

    I moved it to the GetKVM server as I wanted a little more diskspace for testing and thought the extra ram wouldn't hurt.

    In reality, the ram helps a little at times, but the disk io on this node is significantly lower which really pushes up the load on the server during indexing which is a shame.

    The NodeDeploy node gave me about 240MB/s when i tested its io, the GetKVM is at best 70MB/s but typically is nearer 50-60MB/s.

    Given the lower IO when indexing it regularly pushes the load over 1 and at times in the past when i was indexing and backfilling it was hitting 2.

    I think I will probably move it away from this server when I've finished testing a few other things.

    Ideal server would be 1GB ram and 20Gb disk (faster the better). I would say ideal IO would be 150MB/s+ Anything less than 100MB/s will really push your load up if you are indexing a lot of groups.

  • NekkiNekki Moderator

    I was more curious because I'm running it off a KimSufi mKS, and although a tenner a month is a small price to pay really, I do wonder if I couldn't downsize and still keep the same performance...

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • My NodeDeploy node was $48/ year which was ideal. If it wan't for the extra disk space i wanted for testing it would still be there.

    I likely will move it back there or similar host in the near future.

  • @prae5

    One of the restrictions of our OpenVZ based VPS, 2x disk RAID1, which is completely sufficient for the pricing IMHO (And your node is full) but for tasks such as this that require a lot of I/O throughput i can see how it could cause problems such as the ones you have described.

    If you like our service and wish to stay we can also work on a KVM deal for you, these servers run HW RAID10 with BBU and offer a lot more I/O to work with. Just let us know :)

  • Don't get me wrong - I wasn't knocking the service at all. I think its great for the price - i completely understand the restrictions given the price point.

    I'm more than happy with the service - I was just describing the ideal server for indexing.

    I've been working on a few optimisations on my dev server, which I'm looking to push in the near future - it may help. If not will be happy to see what else you can offer.

  • @prae5 said: Don't get me wrong - I wasn't knocking the service at all. I think its great for the price - i completely understand the restrictions given the price point.

    I'm more than happy with the service - I was just describing the ideal server for indexing.

    I've been working on a few optimisations on my dev server, which I'm looking to push in the near future - it may help. If not will be happy to see what else you can offer.

    Thank you for the kind words, we appreciate it, im glad you're happy :)

  • NekkiNekki Moderator

    I must admit, running it on a server with hardware RAID is appealing, right now I'm reliant on offsite backups as if I lose the drive in my mKS I'll just get a fresh one.

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • NekkiNekki Moderator
    edited January 2013

    @srvjap said: San Marino

    http://www.copyright.gov/circs/circ38a.pdf
    http://www.exoticvps.com/

    Unnecessarily expensive considering indexing usenet isn't illegal.

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • @prae5 said: Before running it on this node I tested on a 512Mb node that was SSD based and load was pretty much nothing. I moved it to a NodeDeploy server with 2GB RAM/1GB SWAP and again load was seldom above .2 when indexing and backfilling groups.

    In reality, the ram helps a little at times, but the disk io on this node is significantly lower which really pushes up the load on the server during indexing which is a shame.

    Given the lower IO when indexing it regularly pushes the load over 1 and at times in the past when i was indexing and backfilling it was hitting 2.

    Ideal server would be 1GB ram and 20Gb disk (faster the better). I would say ideal IO would be 150MB/s+ Anything less than 100MB/s will really push your load up if you are indexing a lot of groups.

    Thanks, great info!

    Linux noob willing to learn.

  • NekkiNekki Moderator

    @Freek IMO you might want more disc than that if you're going to backfill quite a lot - I used one of the community provided nzb packs, and the SD TV pack was nearly 15GB on it's own.

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • Member
    @Freek IMO you might want more disc than that if you're going to backfill quite a lot - I used one of the community provided nzb packs, and the SD TV pack was nearly 15GB on it's own.

    That space is only temp required during the backfill process - I used an sshfs temp mount for that.

  • NekkiNekki Moderator

    @prae5 said: That space is only temp required during the backfill process - I used an sshfs temp mount for that.

    I'm not sure I get that - the import is a bunch of NZB's that newznab parses and adds to the index, but the nzbs are still required, so the filesize doesn't change surely?

    Here lies Nekki. He loved massive amounts of storage, K-Pop and calling people cunts.

  • @prae5 We offer disk upgrades you know ;)

    No longer with ND

Sign In or Register to comment.