Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


Rage4 DNS Counts Inflated Queries
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

Rage4 DNS Counts Inflated Queries

BunnySpeedBunnySpeed Member, Host Rep
edited May 2016 in Reviews

To stop things from getting lost under the "review thread" that noone will read, I think opening a new proper thread with a single topic might be more appropriate.

Seems to be the common knowledge now that Rage4 DNS query count seems odd. Rage4 claims this is not the case and tells us they checked this many times, mostly blaming bots and Google's open DNS.

Being all suspicious yesterday because of their poor prorating, I ran a few tests yesterday which you can find here if you want a long read (bottom of page 1): https://www.lowendtalk.com/discussion/81617/rage4-review/p1

I was trying to give them a benefit of the doubt when I was running the first tests, but it slowly started showing that the queries seem to be simply inflated without any real logic. My guess was 5 + 1 (so 6 times).

Updated:

I'm doing more tests, sending a request to: simplest-test-possible.com with no DNS records except for *.simplest-test-possible.com counts as 3 queries.

Sending a request to test.simplest-test-possible.com counts as 6 queries. There is nothing else on there. Since I'm always prefering the option that these aren't just fake queries. Here's what I think you're doing:

You're counting an A type query as a separate request for the SOA record, two NS records, A record AAAA record and CNAME record. Which then appears as 6 queries. (Or something similar)

In any case, this is not how you're suppose to count queries. A DNS query has questions and each question has lookups and then answers. I'm sure 99% of the time one DNS query has one question with multiple lookups and 1-3 results depending on the server config. What you're doing is just counting everything and saying these are queries.

In the first example I assume there's a lookup for a SOA and two NS records since there isn't anything else, counting as 3 requests.

In any case, this is not how you're suppose to count queries. A DNS query has questions and each question has lookups and then answers. I'm sure 99% of the time one DNS query has one question with multiple lookups and 1-3 results depending on the server config. What you're doing is just counting everything and saying these are queries.

Update 2:

It appears that this might only be the case when TTL is low. I don't feel like testing anymore though.

This was then confirmed by @SplitIce who also made an open source testing tool and created a non-existing domain to get as accurate results as possible. You can find the tool at: https://github.com/splitice/Rage4Test

@gbshouse Please join in the conversaion. I don't want to scream and call you a scammer, but as I pointed out many times. Check your query counting, also I feel like you owe a big long explanation that actually doesn't blame everything else. Even though at this point I sincerely doubt there's anything wrong with the code, because something like this would be quite obvious and it would be quite easy for you to say something like: Oh, you were right, it was multiplied by 6. Our bad. That would be a rather poor excuse. Conveniently you always blamed Google DNS for sending 3-5 requests. How fun. Looks just right huh? Except that the tests were done directly.

At this point I'm not sure if I should feel satisfaction of figuring this out, however I am getting quite angry at this point so excuse my "hostility". I never like attacking someone, even if I believe they're doing something wrong and I gave you a benefit of the doubt so many times... Yet it looks like I paid you at least 300€ too much during the past few years for inflating the queries times 6, not to mention many many other, bigger customers. Now convince me otherwise.

Thanked by 2theroyalstudent asf
«134

Comments

  • I've never used Rage4's DNS service so forgive me for maybe being a tad naive when I ask this.

    Do they provide any form of graphing or statistics about "host X did 2.6m queries" or something over a timespan? I know that this could become infeasible quickly, and whether or not they are performing full logging on all requests. (I guess this would inflate quickly!).

    If they are proposing bots as being the cause, then based on the request this should be able to be "skipped" counting wise. If they wish to do this, but I can't see bots hitting your site a significant number of times over the course of a month.

    FOr instance, everytime a whois comes through, a bot would normally crawl your site, too.

    By the looks of that tool it simply just performs a DNS lookup X number of times against Y domain, then I guess you can go back to your dashboard and see these requests counted.

    Whilst this tool exists, and building your own DNS lookup script, it's not going to get us anywhere without @GBSHouse explaining the method of counting eevrything.

    By the sounds of it, I have a gut feeling that somewhere in the Rage4 script, it's doing something like count++; multiple times on the same hit. Again, unsure how these guys log / count so it would be difficult to prove / evaluate otherwise.

    You could always set your domain that you think is getting 1M /hits month to your own DNS, log for a month and then compare, if it's drastically differenent, then you can almost be sure theres an issue with counting.

    Hope this helps.

  • BunnySpeedBunnySpeed Member, Host Rep

    @eastonch My testing was done on a domain that otherwise logged a few 100 requests per day, while @Splitlce registered a non-existing domain with them that can't be hit by bots at all. They also claimed MANY times that they checked their code and that it was written personally and it's perfect.

  • support123support123 Member
    edited May 2016

    well when I was with them my query counts runs in 2 millions and then I started using dnsmadeasy and suddenly it dropped to 400k only
    It was long ago and I am still using dnsmadeeasy and never looked back

    Thanked by 1asf
  • BunnySpeedBunnySpeed Member, Host Rep
    edited May 2016

    @SplitIce and @gbshouse As I don't want to just blindly bash you. I do deel kind of bad for starting all of this, but it had to be done....

    I'm doing more tests, sending a request to: simplest-test-possible.com with no DNS records except for *.simplest-test-possible.com counts as 3 queries.

    Sending a request to test.simplest-test-possible.com counts as 6 queries. There is nothing else on there. Since I'm always prefering the option that these aren't just fake queries. Here's what I think you're doing:

    You're counting an A type query as a separate request for the SOA record, two NS records, A record AAAA record and CNAME record. Which then appears as 6 queries. (Or something similar)

    In the first example I assume there's a lookup for a SOA and two NS records since there isn't anything else, counting as 3 requests.

    In any case, this is not how you're suppose to count queries. A DNS query has questions and each question has lookups and then answers. I'm sure 99% of the time one DNS query has one question with multiple lookups and 1-3 results depending on the server config. What you're doing is just counting everything and saying these are queries.

  • hoczajhoczaj Member

    Hm maybe you're right. It could have happened that instead of counter++ it became counter += records.count()

    I did some test too, your theory seems right.

    I wish it would be some open source code. Actually I like when someone go for open source. We should note that open source does not equal with Free. You can still license it properly.

  • BunnySpeedBunnySpeed Member, Host Rep
    edited May 2016

    @hoczaj I think it's counting lookups, not records or even results.
    This is what's returned:

    This being said, I don't care what lookups they are doing, the pricing clearly states this is per DNS query. Then they should say that the pricing is per lookup, not per query. So there's something indeed wrong in the code.

  • I think @gbshouse mentioned that metering is implemented at the database level. When a DNS server gets a request it might be reading all the DNS records for caching and gets counted 6 times, even if you asked only for the A record.

    You might also get some overcounting because of Anycast where multiple locations get and respond to the same query. Maybe you can decrease IP_TTL with setsockopt in your request, so that only one location responds - to verify if this is an issue.

    What I don't understand is if your query is getting cached by the DNS server, then repeated requests should never have hit the database once the caches are warm. So unless the cache is broken, your multiplication factor (6?) would never have shown up for something like 1k or 10k requests.

  • BunnySpeedBunnySpeed Member, Host Rep
    edited May 2016

    @rincewind said:
    I think @gbshouse mentioned that metering is implemented at the database level. When a DNS server gets a request it might be reading all the DNS records for caching and gets counted 6 times, even if you asked only for the A record.

    You might also get some overcounting because of Anycast where multiple locations get and respond to the same query. Maybe you can decrease IP_TTL with setsockopt in your request, so that only one location responds - to verify if this is an issue.

    What I don't understand is if your query is getting cached by the DNS server, then repeated requests should never have hit the database once the caches are warm. So unless the cache is broken, your multiplication factor (6?) would never have shown up for something like 1k or 10k requests.

    Well, that's just really poor design on their end and it's technically a lie since they claim queries are counted and these are not queries. But I don't care about that. What I cound is the end result.

    The magic of their service is of course that they suggest a 5 second TTL that doesn't cache. So it means 6 requests are always pulled from the database or something. Either this, or caching is broken.

    What's bothering me that it's clearly a problem, yet they don't want to do anything about it. It's like me saying, yeah we count bandwidth differently. Inbound bandwidth is counted as well and our caching sucks so we're always just pulling stuff from your server and you're getting a double bill. Except it's not double but 6 times the amount in this case.

  • ZappieZappie Member, Host Rep, LIR

    rincewind said: When a DNS server gets a request it might be reading all the DNS records for caching and gets counted 6 times, even if you asked only for the A record

    Sounds weird if that is the case. On that logic it means say you have 100 TXT records for a domain, when someone submits a single TXT and 100 TXT records gets returned then it counts at 100 queries? If that is the case maybe they should change the terminalogy from
    $X for Y Queries to $X per Y Returned records

    rincewind said: You might also get some overcounting because of Anycast where multiple locations get and respond to the same query

    I think you are getting mixed up with Mutlicast.
    Anycast will go to a single (shortest path) location.

    Thanked by 1Kris
  • Zappie said: Multicast

    Ah yes. My bad.

    I think the initial overcounting while warming the caches would have been Ok if the caches handled the rest. Maybe a design choice for handling peak loads.

  • KrisKris Member

    We all asked the questions. I even raised EVERY TTL to the maximum allowed and no changes.

    35 domains on DNS Made Easy doesn't go over 1.2 million queries per month, and 5-10 domains have TTL's of 5-60...

    As Ms. Cleo said, the cards don't lie...

    http://www.dnsmadeeasy.com/pricing/

    They were #1 by speed in April 2016, and many other months : http://www.solvedns.com/dns-comparison/2016/04

    Used to use CloudNS but 100ms+ is not acceptable. DNSMadeEasy is 'set it and forget it' if you just want proper Anycast DNS done right, sans oddities, outliers, and a headache about how a domain with almost no traffic kept hitting 1 million queries per month.

    Uneven AS-PATHs makes for a hard way to properly distribute Anycast and you end up with the below, unpredictable routing


    Rage 4 MTRs : (NL -> Atlanta, GA) (Newark, NJ -> Amsterdam, NL)
    https://pulse.turbobytes.com/results/5729e990ecbe404fe6000ddd/

    DNSMadeEasy MTRs:
    https://pulse.turbobytes.com/results/5729ea24ecbe404fe6000ddf/


    Make up your own mind who you want to be paying for Anycast DNS, but MTR's from 100+ locations paint a picture of an uneven AS-PATH.

  • gbshousegbshouse Member, Host Rep

    I'll try to address above

    @Kris - noone said that as-paths must be even, especially when you use various upstream, in our case we use in-house build AI for route optimization, unfortunately since we are in process of rebuilding our network it's still in learning mode

    BunnySpeed said: The magic of their service is of course that they suggest a 5 second TTL that doesn't cache. So it means 6 requests are always pulled from the database or something. Either this, or caching is broken.

    I agree that our cache is not perfect. Yes, we do count queries on backend level. Since we are using customized version of PowerDNS you can analyze their sources and see how it's handled.

    BunnySpeed said: What's bothering me that it's clearly a problem, yet they don't want to do anything about it.

    That's not true. We want to make our customers happy and to achieve this we want to make everything as painless and transparent as possible. I'm just talking to one of our partners regarding independent code review. We've also started the work related to more transparent usage statistics.

  • BunnySpeedBunnySpeed Member, Host Rep
    edited May 2016

    @gbshouse said:
    I'll try to address above

    Can you tell us if the 5 second TTL queries are cached or do they always go to the backend, generating 6 requests every time?

    If you have poor design, that doesn't justify counting multiple queries per request. This has been addressed many times over the years and it's still happening and that's why I said you don't want to do anything about it.

    I don't want to call you a scammer since you're clearly not scamming anyone, but you do have inflated queries. For end users it's not important why. They're just there.

    This drove me away personally because the cost was unsustainable which is also the reason why I went and tried really hard to find out what was actually going on. I am aware you're working really hard on your product and I respect that. In fact I feel kind of bad for all of this, however you can't do things like this to your customers just because of bad design.

    Thanked by 1asf
  • gbshousegbshouse Member, Host Rep

    BunnySpeed said: Can you tell us if the 5 second TTL queries are cached or do they always go to the backend, generating 6 requests every time?

    All records with TTL <=5 are served directly from backend and not cached. No, they are not generating 6 requests every time. PowerDNS query flow is (from the top of my head) for zone example.com query test.example.com: SOA for test.example.com (not counted), SOA for example.com (not counted), NS for test.example.com (not counted), ANY for test.example.com (count 1), filter correct record type from last result.

  • BunnySpeedBunnySpeed Member, Host Rep
    edited May 2016

    @gbshouse said:

    BunnySpeed said: Can you tell us if the 5 second TTL queries are cached or do they always go to the backend, generating 6 requests every time?

    No, they are not generating 6 requests every time.

    How come they do in the tests though? Any idea? You can't blame the cache since you said they don't cache.

    Ever since we switched off your system the query count is approximately 6 times smaller so it's not just the tests either.

  • KrisKris Member

    gbshouse said: @Kris - noone said that as-paths must be even, especially when you use various upstream, in our case we use in-house build AI for route optimization, unfortunately since we are in process of rebuilding our network it's still in learning mode

    It's just best practice to do such. And if you aren't using an even AS-PATH, you need some serious communities pushed upstream or at least some prepending done in your BGP configuration so Newark doesn't cross the ocean, NL doesn't route to the US, etc.

  • gbshousegbshouse Member, Host Rep

    @Kris - just remember that we optimize our network against public DNS servers and not end user

    @BunnySpeed - the only place which can potentially, I repeat potentially, do something "nasty" is the software which gathers statistics from the servers.

  • gbshouse said: All records with TTL <=5 are served directly from backend and not cached

    Why would server-side caching depend on the TTL value? It should only impact DNS clients, causing them to poll more often.

    On the server side, your caches would invalidate only if a DNS record actually changed. Your DNS server-side cache could subscribe to entries in the backend, and DNS record changes would trigger a publish/invalidate to the server caches. With your current design I think your DB must be taking a beating.

  • vldvld Member

    I remember a bug with their geo dns and caching, which gbshouse did not want to admit to. If you hit one of their PoPs with a EDNS0 query containt a client subnet it will get cached on the server side. So if within the TTL another request comes, it will ignore the client subnet and return the cached response.

    This made their geo implementation pretty much useless when you have more than 1 request per second hitting their PoPs, even with a TTL of <= 5.

  • raindog308raindog308 Administrator, Veteran

    rincewind said: I think @gbshouse mentioned that metering is implemented at the database level.

    I'm kinda curious what the "database" part of this is. Is this something like MySQL?

    No idea what Rage4 runs but I imagine their engineering problem is that they want to answer queries but also log for billing purposes. I'm don't know what bind or nsd does for logging but having them actually ask the DB for DNS records seems very daffy to me. Having the DNS server log its traffic, maybe to MySQL, makes sense...but storing the .db records in MySQL is...weird.

    I can see why someone might do that from the perspective of a web app that lets people update/change DNS but that's not a good design. The web app should parse the .db record, present it, take the changes, and queue a new .db record. Having bind ask MySQL "what answer should I give" has got to be a hideous performance hit, as well as very error-prone as some people seem to be experiencing...

    Random first-coffee-of-the-day thoughts.

  • gbshousegbshouse Member, Host Rep

    @vld - that was fixed long time ago

  • vldvld Member

    gbshouse said: @vld - that was fixed long time ago

    That's the point I'm trying to make. Even tho I reported the bug multiple times, and every time you refused to admit the bug existed, somehow it eventually got fixed.

  • gbshousegbshouse Member, Host Rep

    @raindog308 - we use Percona MySql and it works fine even with large workloads. It's just a matter of fine tuning of both configuration and queries. We do not use flat files for zone storage.

  • raindog308raindog308 Administrator, Veteran

    gbshouse said: @raindog308 - we use Percona MySql and it works fine even with large workloads. It's just a matter of fine tuning of both configuration and queries. We do not use flat files for zone storage.

    Interesting...I would not have expected that (I'd think the DB round-time would horribly slow down DNS), but obviously you built it so that part works.

  • HackedServerHackedServer Member
    edited May 2016

    @raindog308 said:
    Having the DNS server log its traffic, maybe to MySQL, makes sense...but storing the .db records in MySQL is...weird.

    I believe PowerDNS backed by MySQL is pretty common.
    https://doc.powerdns.com/md/authoritative/#backends

    Many things, like PowerDNS GUIs and Virtualizor can connect to a PowerDNS DB to manage records, and then PowerDNS serves from that database.

    If Rage4 did "fix" this, I imagine it would take a chunk out of their profits. What's worse, some unhappy people on LET or cutting your income by potentially 4-6 times?

  • bobbybobby Member
    edited May 2016

    No issue with the service at all, its excellent, but my impression is that query count is inflated compared to others (edit: I have a prometeus account so don't care, but I did notice after changing 2 domains)

  • praveenpraveen Member
    edited May 2016

    @bobby said:
    (edit: I have a prometeus account so don't care, but I did notice after changing 2 domains)

    if you look on your cp you can see that it also has a limit of 100000000 Free usage

  • BunnySpeedBunnySpeed Member, Host Rep
    edited May 2016

    @HackedServer said:
    If Rage4 did "fix" this, I imagine it would take a chunk out of their profits. What's worse, some unhappy people on LET or cutting your income by potentially 4-6 times?

    I'm thinking the same thing. Can't really blame them. It would probably be a huge blow.

  • gbshousegbshouse Member, Host Rep

    @BunnySpeed, @HackedServer - unfortunately no, the users on PAYG are ~ 5% of our profit, and to be honest LET is not our primary target. The main source of our income are large and very large customers which use flat fee packages and don't care about usage.

Sign In or Register to comment.