Rage4 DNS Counts Inflated Queries
New on LowEndTalk? Please read our 'Community Rules' by clicking on it in the right menu!

Rage4 DNS Counts Inflated Queries

BunnySpeedBunnySpeed Member, Provider
edited May 2016 in Reviews

To stop things from getting lost under the "review thread" that noone will read, I think opening a new proper thread with a single topic might be more appropriate.

Seems to be the common knowledge now that Rage4 DNS query count seems odd. Rage4 claims this is not the case and tells us they checked this many times, mostly blaming bots and Google's open DNS.

Being all suspicious yesterday because of their poor prorating, I ran a few tests yesterday which you can find here if you want a long read (bottom of page 1): https://www.lowendtalk.com/discussion/81617/rage4-review/p1

I was trying to give them a benefit of the doubt when I was running the first tests, but it slowly started showing that the queries seem to be simply inflated without any real logic. My guess was 5 + 1 (so 6 times).

Updated:

I'm doing more tests, sending a request to: simplest-test-possible.com with no DNS records except for *.simplest-test-possible.com counts as 3 queries.

Sending a request to test.simplest-test-possible.com counts as 6 queries. There is nothing else on there. Since I'm always prefering the option that these aren't just fake queries. Here's what I think you're doing:

You're counting an A type query as a separate request for the SOA record, two NS records, A record AAAA record and CNAME record. Which then appears as 6 queries. (Or something similar)

In any case, this is not how you're suppose to count queries. A DNS query has questions and each question has lookups and then answers. I'm sure 99% of the time one DNS query has one question with multiple lookups and 1-3 results depending on the server config. What you're doing is just counting everything and saying these are queries.

In the first example I assume there's a lookup for a SOA and two NS records since there isn't anything else, counting as 3 requests.

In any case, this is not how you're suppose to count queries. A DNS query has questions and each question has lookups and then answers. I'm sure 99% of the time one DNS query has one question with multiple lookups and 1-3 results depending on the server config. What you're doing is just counting everything and saying these are queries.

Update 2:

It appears that this might only be the case when TTL is low. I don't feel like testing anymore though.

This was then confirmed by @SplitIce who also made an open source testing tool and created a non-existing domain to get as accurate results as possible. You can find the tool at: https://github.com/splitice/Rage4Test

@gbshouse Please join in the conversaion. I don't want to scream and call you a scammer, but as I pointed out many times. Check your query counting, also I feel like you owe a big long explanation that actually doesn't blame everything else. Even though at this point I sincerely doubt there's anything wrong with the code, because something like this would be quite obvious and it would be quite easy for you to say something like: Oh, you were right, it was multiplied by 6. Our bad. That would be a rather poor excuse. Conveniently you always blamed Google DNS for sending 3-5 requests. How fun. Looks just right huh? Except that the tests were done directly.

At this point I'm not sure if I should feel satisfaction of figuring this out, however I am getting quite angry at this point so excuse my "hostility". I never like attacking someone, even if I believe they're doing something wrong and I gave you a benefit of the doubt so many times... Yet it looks like I paid you at least 300€ too much during the past few years for inflating the queries times 6, not to mention many many other, bigger customers. Now convince me otherwise.

Thanked by 2theroyalstudent asf

BunnyCDN - Make your website as fast as a bunny

«13

Comments

  • I've never used Rage4's DNS service so forgive me for maybe being a tad naive when I ask this.

    Do they provide any form of graphing or statistics about "host X did 2.6m queries" or something over a timespan? I know that this could become infeasible quickly, and whether or not they are performing full logging on all requests. (I guess this would inflate quickly!).

    If they are proposing bots as being the cause, then based on the request this should be able to be "skipped" counting wise. If they wish to do this, but I can't see bots hitting your site a significant number of times over the course of a month.

    FOr instance, everytime a whois comes through, a bot would normally crawl your site, too.

    By the looks of that tool it simply just performs a DNS lookup X number of times against Y domain, then I guess you can go back to your dashboard and see these requests counted.

    Whilst this tool exists, and building your own DNS lookup script, it's not going to get us anywhere without @GBSHouse explaining the method of counting eevrything.

    By the sounds of it, I have a gut feeling that somewhere in the Rage4 script, it's doing something like count++; multiple times on the same hit. Again, unsure how these guys log / count so it would be difficult to prove / evaluate otherwise.

    You could always set your domain that you think is getting 1M /hits month to your own DNS, log for a month and then compare, if it's drastically differenent, then you can almost be sure theres an issue with counting.

    Hope this helps.

    Security Consultant

  • BunnySpeedBunnySpeed Member, Provider

    @eastonch My testing was done on a domain that otherwise logged a few 100 requests per day, while @Splitlce registered a non-existing domain with them that can't be hit by bots at all. They also claimed MANY times that they checked their code and that it was written personally and it's perfect.

    BunnyCDN - Make your website as fast as a bunny

  • support123support123 Member
    edited May 2016

    well when I was with them my query counts runs in 2 millions and then I started using dnsmadeasy and suddenly it dropped to 400k only
    It was long ago and I am still using dnsmadeeasy and never looked back

    Thanked by 1asf

    I am not Rick

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @SplitIce and @gbshouse As I don't want to just blindly bash you. I do deel kind of bad for starting all of this, but it had to be done....

    I'm doing more tests, sending a request to: simplest-test-possible.com with no DNS records except for *.simplest-test-possible.com counts as 3 queries.

    Sending a request to test.simplest-test-possible.com counts as 6 queries. There is nothing else on there. Since I'm always prefering the option that these aren't just fake queries. Here's what I think you're doing:

    You're counting an A type query as a separate request for the SOA record, two NS records, A record AAAA record and CNAME record. Which then appears as 6 queries. (Or something similar)

    In the first example I assume there's a lookup for a SOA and two NS records since there isn't anything else, counting as 3 requests.

    In any case, this is not how you're suppose to count queries. A DNS query has questions and each question has lookups and then answers. I'm sure 99% of the time one DNS query has one question with multiple lookups and 1-3 results depending on the server config. What you're doing is just counting everything and saying these are queries.

    BunnyCDN - Make your website as fast as a bunny

  • hoczajhoczaj Member

    Hm maybe you're right. It could have happened that instead of counter++ it became counter += records.count()

    I did some test too, your theory seems right.

    I wish it would be some open source code. Actually I like when someone go for open source. We should note that open source does not equal with Free. You can still license it properly.

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @hoczaj I think it's counting lookups, not records or even results.
    This is what's returned:

    This being said, I don't care what lookups they are doing, the pricing clearly states this is per DNS query. Then they should say that the pricing is per lookup, not per query. So there's something indeed wrong in the code.

    BunnyCDN - Make your website as fast as a bunny

  • I think @gbshouse mentioned that metering is implemented at the database level. When a DNS server gets a request it might be reading all the DNS records for caching and gets counted 6 times, even if you asked only for the A record.

    You might also get some overcounting because of Anycast where multiple locations get and respond to the same query. Maybe you can decrease IP_TTL with setsockopt in your request, so that only one location responds - to verify if this is an issue.

    What I don't understand is if your query is getting cached by the DNS server, then repeated requests should never have hit the database once the caches are warm. So unless the cache is broken, your multiplication factor (6?) would never have shown up for something like 1k or 10k requests.

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @rincewind said:
    I think @gbshouse mentioned that metering is implemented at the database level. When a DNS server gets a request it might be reading all the DNS records for caching and gets counted 6 times, even if you asked only for the A record.

    You might also get some overcounting because of Anycast where multiple locations get and respond to the same query. Maybe you can decrease IP_TTL with setsockopt in your request, so that only one location responds - to verify if this is an issue.

    What I don't understand is if your query is getting cached by the DNS server, then repeated requests should never have hit the database once the caches are warm. So unless the cache is broken, your multiplication factor (6?) would never have shown up for something like 1k or 10k requests.

    Well, that's just really poor design on their end and it's technically a lie since they claim queries are counted and these are not queries. But I don't care about that. What I cound is the end result.

    The magic of their service is of course that they suggest a 5 second TTL that doesn't cache. So it means 6 requests are always pulled from the database or something. Either this, or caching is broken.

    What's bothering me that it's clearly a problem, yet they don't want to do anything about it. It's like me saying, yeah we count bandwidth differently. Inbound bandwidth is counted as well and our caching sucks so we're always just pulling stuff from your server and you're getting a double bill. Except it's not double but 6 times the amount in this case.

    BunnyCDN - Make your website as fast as a bunny

  • ZappieZappie Member, Provider

    rincewind said: When a DNS server gets a request it might be reading all the DNS records for caching and gets counted 6 times, even if you asked only for the A record

    Sounds weird if that is the case. On that logic it means say you have 100 TXT records for a domain, when someone submits a single TXT and 100 TXT records gets returned then it counts at 100 queries? If that is the case maybe they should change the terminalogy from
    $X for Y Queries to $X per Y Returned records

    rincewind said: You might also get some overcounting because of Anycast where multiple locations get and respond to the same query

    I think you are getting mixed up with Mutlicast.
    Anycast will go to a single (shortest path) location.

    Thanked by 1Kris

    Cheap VPS in Auckland, New Zealand and Johannesburg, South Africa . We take hosting seriously.

  • Zappie said: Multicast

    Ah yes. My bad.

    I think the initial overcounting while warming the caches would have been Ok if the caches handled the rest. Maybe a design choice for handling peak loads.

  • KrisKris Member

    We all asked the questions. I even raised EVERY TTL to the maximum allowed and no changes.

    35 domains on DNS Made Easy doesn't go over 1.2 million queries per month, and 5-10 domains have TTL's of 5-60...

    As Ms. Cleo said, the cards don't lie...

    http://www.dnsmadeeasy.com/pricing/

    They were #1 by speed in April 2016, and many other months : http://www.solvedns.com/dns-comparison/2016/04

    Used to use CloudNS but 100ms+ is not acceptable. DNSMadeEasy is 'set it and forget it' if you just want proper Anycast DNS done right, sans oddities, outliers, and a headache about how a domain with almost no traffic kept hitting 1 million queries per month.

    Uneven AS-PATHs makes for a hard way to properly distribute Anycast and you end up with the below, unpredictable routing


    Rage 4 MTRs : (NL -> Atlanta, GA) (Newark, NJ -> Amsterdam, NL)
    https://pulse.turbobytes.com/results/5729e990ecbe404fe6000ddd/

    DNSMadeEasy MTRs:
    https://pulse.turbobytes.com/results/5729ea24ecbe404fe6000ddf/


    Make up your own mind who you want to be paying for Anycast DNS, but MTR's from 100+ locations paint a picture of an uneven AS-PATH.

  • gbshousegbshouse Member, Provider

    I'll try to address above

    @Kris - noone said that as-paths must be even, especially when you use various upstream, in our case we use in-house build AI for route optimization, unfortunately since we are in process of rebuilding our network it's still in learning mode

    BunnySpeed said: The magic of their service is of course that they suggest a 5 second TTL that doesn't cache. So it means 6 requests are always pulled from the database or something. Either this, or caching is broken.

    I agree that our cache is not perfect. Yes, we do count queries on backend level. Since we are using customized version of PowerDNS you can analyze their sources and see how it's handled.

    BunnySpeed said: What's bothering me that it's clearly a problem, yet they don't want to do anything about it.

    That's not true. We want to make our customers happy and to achieve this we want to make everything as painless and transparent as possible. I'm just talking to one of our partners regarding independent code review. We've also started the work related to more transparent usage statistics.

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @gbshouse said:
    I'll try to address above

    Can you tell us if the 5 second TTL queries are cached or do they always go to the backend, generating 6 requests every time?

    If you have poor design, that doesn't justify counting multiple queries per request. This has been addressed many times over the years and it's still happening and that's why I said you don't want to do anything about it.

    I don't want to call you a scammer since you're clearly not scamming anyone, but you do have inflated queries. For end users it's not important why. They're just there.

    This drove me away personally because the cost was unsustainable which is also the reason why I went and tried really hard to find out what was actually going on. I am aware you're working really hard on your product and I respect that. In fact I feel kind of bad for all of this, however you can't do things like this to your customers just because of bad design.

    Thanked by 1asf

    BunnyCDN - Make your website as fast as a bunny

  • gbshousegbshouse Member, Provider

    BunnySpeed said: Can you tell us if the 5 second TTL queries are cached or do they always go to the backend, generating 6 requests every time?

    All records with TTL <=5 are served directly from backend and not cached. No, they are not generating 6 requests every time. PowerDNS query flow is (from the top of my head) for zone example.com query test.example.com: SOA for test.example.com (not counted), SOA for example.com (not counted), NS for test.example.com (not counted), ANY for test.example.com (count 1), filter correct record type from last result.

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @gbshouse said:

    BunnySpeed said: Can you tell us if the 5 second TTL queries are cached or do they always go to the backend, generating 6 requests every time?

    No, they are not generating 6 requests every time.

    How come they do in the tests though? Any idea? You can't blame the cache since you said they don't cache.

    Ever since we switched off your system the query count is approximately 6 times smaller so it's not just the tests either.

    BunnyCDN - Make your website as fast as a bunny

  • KrisKris Member

    gbshouse said: @Kris - noone said that as-paths must be even, especially when you use various upstream, in our case we use in-house build AI for route optimization, unfortunately since we are in process of rebuilding our network it's still in learning mode

    It's just best practice to do such. And if you aren't using an even AS-PATH, you need some serious communities pushed upstream or at least some prepending done in your BGP configuration so Newark doesn't cross the ocean, NL doesn't route to the US, etc.

  • gbshousegbshouse Member, Provider

    @Kris - just remember that we optimize our network against public DNS servers and not end user

    @BunnySpeed - the only place which can potentially, I repeat potentially, do something "nasty" is the software which gathers statistics from the servers.

  • gbshouse said: All records with TTL <=5 are served directly from backend and not cached

    Why would server-side caching depend on the TTL value? It should only impact DNS clients, causing them to poll more often.

    On the server side, your caches would invalidate only if a DNS record actually changed. Your DNS server-side cache could subscribe to entries in the backend, and DNS record changes would trigger a publish/invalidate to the server caches. With your current design I think your DB must be taking a beating.

  • vldvld Member

    I remember a bug with their geo dns and caching, which gbshouse did not want to admit to. If you hit one of their PoPs with a EDNS0 query containt a client subnet it will get cached on the server side. So if within the TTL another request comes, it will ignore the client subnet and return the cached response.

    This made their geo implementation pretty much useless when you have more than 1 request per second hitting their PoPs, even with a TTL of <= 5.

  • raindog308raindog308 Moderator

    rincewind said: I think @gbshouse mentioned that metering is implemented at the database level.

    I'm kinda curious what the "database" part of this is. Is this something like MySQL?

    No idea what Rage4 runs but I imagine their engineering problem is that they want to answer queries but also log for billing purposes. I'm don't know what bind or nsd does for logging but having them actually ask the DB for DNS records seems very daffy to me. Having the DNS server log its traffic, maybe to MySQL, makes sense...but storing the .db records in MySQL is...weird.

    I can see why someone might do that from the perspective of a web app that lets people update/change DNS but that's not a good design. The web app should parse the .db record, present it, take the changes, and queue a new .db record. Having bind ask MySQL "what answer should I give" has got to be a hideous performance hit, as well as very error-prone as some people seem to be experiencing...

    Random first-coffee-of-the-day thoughts.

    My Advice: VPS Advice

    For LET support, please click here.

  • vldvld Member

    gbshouse said: @vld - that was fixed long time ago

    That's the point I'm trying to make. Even tho I reported the bug multiple times, and every time you refused to admit the bug existed, somehow it eventually got fixed.

  • gbshousegbshouse Member, Provider

    @raindog308 - we use Percona MySql and it works fine even with large workloads. It's just a matter of fine tuning of both configuration and queries. We do not use flat files for zone storage.

  • raindog308raindog308 Moderator

    gbshouse said: @raindog308 - we use Percona MySql and it works fine even with large workloads. It's just a matter of fine tuning of both configuration and queries. We do not use flat files for zone storage.

    Interesting...I would not have expected that (I'd think the DB round-time would horribly slow down DNS), but obviously you built it so that part works.

    My Advice: VPS Advice

    For LET support, please click here.

  • HackedServerHackedServer Member
    edited May 2016

    @raindog308 said:
    Having the DNS server log its traffic, maybe to MySQL, makes sense...but storing the .db records in MySQL is...weird.

    I believe PowerDNS backed by MySQL is pretty common.
    https://doc.powerdns.com/md/authoritative/#backends

    Many things, like PowerDNS GUIs and Virtualizor can connect to a PowerDNS DB to manage records, and then PowerDNS serves from that database.

    If Rage4 did "fix" this, I imagine it would take a chunk out of their profits. What's worse, some unhappy people on LET or cutting your income by potentially 4-6 times?

  • bobbybobby Member
    edited May 2016

    No issue with the service at all, its excellent, but my impression is that query count is inflated compared to others (edit: I have a prometeus account so don't care, but I did notice after changing 2 domains)

  • praveenpraveen Member
    edited May 2016

    @bobby said:
    (edit: I have a prometeus account so don't care, but I did notice after changing 2 domains)

    if you look on your cp you can see that it also has a limit of 100000000 Free usage

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @HackedServer said:
    If Rage4 did "fix" this, I imagine it would take a chunk out of their profits. What's worse, some unhappy people on LET or cutting your income by potentially 4-6 times?

    I'm thinking the same thing. Can't really blame them. It would probably be a huge blow.

    BunnyCDN - Make your website as fast as a bunny

  • gbshousegbshouse Member, Provider

    @BunnySpeed, @HackedServer - unfortunately no, the users on PAYG are ~ 5% of our profit, and to be honest LET is not our primary target. The main source of our income are large and very large customers which use flat fee packages and don't care about usage.

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    I think this concludes the story lol :D

    Thanked by 2netomx Nyr

    BunnyCDN - Make your website as fast as a bunny

  • tr1ckytr1cky Member

    Was pretty much expected that nothing happens.

    tsdns.io - free, redundant, DDoS-protected TSDNS

  • tommytommy Member

    BunnySpeed said: I think this concludes the story lol :D

    Have fun :D

    Thanked by 1netomx

    Let's bet which dot-name will collapse first ;)

  • mik997mik997 Member

    lemme guess .. your Rage4 account got disabled ;P

    which provider are you using for geo DNS now out of curiosity?

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @mik997 said:
    lemme guess .. your Rage4 account got disabled ;P

    which provider are you using for geo DNS now out of curiosity?

    It was, I think. The API suddenly stopped working and I can't log in anymore... Fun. Thankfully nothing was running on it anymore so it doesn't really matter.

    For personal stuff I use Amazon Route 53 just because it's pay as you go and my stuff is already there, otherwise I'd go with DNS Made Easy. I might switch there in the future actually. For the CDN we're using a custom solution now.

    Thanked by 1mik997

    BunnyCDN - Make your website as fast as a bunny

  • ZappieZappie Member, Provider

    mik997 said: which provider are you using for geo DNS now out of curiosity?

    I would highly recommend a self hosted solution with gDNSd. It can pretty much do the same (and more actually) in terms of failover, geodns and loadbalancing as what rage4 can do.

    Downside is you will have to maintain it yourself on your servers (which I assume wont be on an anycast network)

    Thanked by 2mik997 geekalot

    Cheap VPS in Auckland, New Zealand and Johannesburg, South Africa . We take hosting seriously.

  • BunnySpeedBunnySpeed Member, Provider
    edited May 2016

    @Zappie said:

    mik997 said: which provider are you using for geo DNS now out of curiosity?

    I would highly recommend a self hosted solution with gDNSd. It can pretty much do the same (and more actually) in terms of failover, geodns and loadbalancing as what rage4 can do.

    Downside is you will have to maintain it yourself on your servers (which I assume wont be on an anycast network)

    We're currently running 6 VMs on BuyVM, 3x OpenVZ and 3x KVM. It's not ideal but not that bad actually. I did some tests a few days ago and strangely it seems faster than CDN77 DNS o_O. But we're looking to upgrade to add Singapore in the near future.

    We also run custom software that allows great smart balancing / routing :)

    BunnyCDN - Make your website as fast as a bunny

  • ButtersButters Member

    Rule #1

    Never use any service where william is involved like Rage4, Airvpn or ...

    Thanked by 1boernd

    Lu Lu Lu, I've got some apples. Lu Lu Lu you've got some too.

  • These guys have know about the issue since day one.
    The time to have corrected it would have been early on.

    If they admit to an error in the calculations at this point in time, they would have to reimburse all their clients.

  • tr1ckytr1cky Member
    edited May 2016

    What was that newcomer-alternative that was mentioned somewhere which is free right now?

    Nevermind, already found: https://zilore.com/en

    Thanked by 1theroyalstudent

    tsdns.io - free, redundant, DDoS-protected TSDNS

  • vldvld Member

    @Butters said:
    Rule #1

    Never use any service where william is involved like Rage4, Airvpn or ...

    Ugh, what?

  • filefile Member

    Is anyone actively using Zilore who can comment on them and their service?

    Core Asterisk Developer and Hater of Fax Machines

  • tr1ckytr1cky Member

    @file said:
    Is anyone actively using Zilore who can comment on them and their service?

    I created an account today, but the lowest TTLs you can set are 60 seconds, that's too much for my failover stuff.

    Also seems like they do not support CNAME failover or wildcard failover, so I'm out.

    tsdns.io - free, redundant, DDoS-protected TSDNS

  • gbshousegbshouse Member, Provider

    @Butters William is not involved in Rage4, just minor shareholder

  • He is unable to login

    I am not Rick

  • gbshousegbshouse Member, Provider

    @tr1cky - we are waiting for final audit results but as far as I know everything is ok.

    From the news - we plan to move from pay-as-you-go to flat fee only packages, also we will no longer offer free accounts.

  • @BunnySpeed said:
    I think this concludes the story lol :D

    Tried resetting the password ? or username not valid even ?

    I am not Rick

  • tr1ckytr1cky Member

    gbshouse said: also we will no longer offer free accounts.

    You mean prometeus accounts?

    tsdns.io - free, redundant, DDoS-protected TSDNS

  • NeoonNeoon Member

    @gbshouse said:
    @tr1cky - we are waiting for final audit results but as far as I know everything is ok.

    From the news - we plan to move from pay-as-you-go to flat fee only packages, also we will no longer offer free accounts.

    ...... So you will also remove the Prometheus Acccount offer?

Sign In or Register to comment.