All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
How to set up anycast IP address (anycast DNS)?
Having 1000+ domains in play, time has come to set up own nameservers (using NSD). All 1000+ websites will be on 2 webservers (backends) in the US with Varnish frontends, initially in 7 locations (Frankfurt, Atlanta, Phoenix, Singapore, Pune, Shanghai and Sydney). Hence my interest in setting up anycast DNS for optimum performance. I would like to start with matching nameservers in above locations.
Problem: while I understand pretty well how anycast works, I have been unable to find any practical guide on how to set up the anycast IPv4 needed to make this work, specifically:
how to forward the IP address from one location (e.g. Atlanta KVM = primary NS) to the other 6 KVM boxes (secondary NS's), hosted with different providers?
and how to configure return paths from each NS?
I will be grateful for any pointers or links. Any critique or ideas concerning the overall setup are also welcome 
PS: sorry, not interested in existing hosted DNS services. I would like to have one all my own, especially since it's justified economically 
Comments
Ill let the folks at Carnegie Mellon and NANOg explain it.
https://nanog.org/meetings/nanog29/presentations/miller.pdf
Not going to work unless you have BGP at all locations and your own /24 (or rather multiple).
Source: I run this for a living with Rage4
It's also worth mentioning that it may be best if the Subnets are announced under the same ASN in all locations and preferably with the same peers. This way it's easier to optimize paths and redirect incorrect routing
That's understandable. You may need to make sure to have a rather healthy budget
Rage4 :P
I agree with William.
You mention a couple of (KVM) VPS, which is a nice DNS solution, but which also hints at you having very little influence in the game.
Don't forget that Anycast is a routing scheme!
And why would you want to do that in the first place? What is it you strive for?
Typical reasons (and probably also true for you) are to do with "I want good global reachability".
Turn it upside down for a moment. Most carriers have an interest that handsomely matches yours, albeit for other reasons (cost). So usually you can assume that the carriers also have the interest to not have packets travel needlessly.
Finally you can get something quite close to what you (probably) want by evaluating the requestors address. (Overly) simple example: A query comes from Europe? Fine, tell them a European address. Next one from India? OK, give them an Indian (or uae, or ...) address as response, aso.
(Yes, I know, the requestor will often be an ISPs recursive server which may be at the other end of the world. Usually, however, they aren't too far apart and, hey, Anycast isn't a clean 100% solution either).
Good luck for your project!
Thanks! And thanks for your take on this. As a matter of fact, I'm evaluating another option: BIND Views, which pretty much works out to what you suggest. As for the global IP anycast, before diving all in, I'm planning on testing my setup in the 3 existing "anycast" locations offered by BuyVM.
EDIT: to clarify my reasons:
(1) I need my own DNS with custom CP + DB backend + tools to manage the 1000+ domains, including automated, dynamic record updates
(2) I need anycast DNS or another corresponding solution to intelligently route traffic to my closest Varnish frontend.
You mean BuyVM
Honestly, A lot of AU->Asia traffic goes through the US anyway, like it or not.
Francisco
Of course! My apologies for that! Corrected.
I'm aware. Hence one of my Varnish frontends planned for Sydney
@aglodek
Unless your sites for those 1000+ domains are very primitive you might want to focus on another and more ugly problem: database.
You may play all funny games but as soon as you go dynamic (site-wise) and most probably use a database you have a problem because DBs don't care about AnyCast - but you (should) care about consistency.
Using front caches like Varnish is nice and dandy but won't be happy about dynamic sites. So you might want to consider a more elaborate solution (a smarter "Varnish") with a smart DB management layer.
Just saying.
@bsdguy: way ahead of you on that score, I think
Webservers (backends) running Drupal with offloaded MariaDB/Galera cluster (5 nodes).
EDIT: and yes, conveniently, initially most of the websites will be pretty primitive with few writes to DB, mostly (90%) reads.
aglodek
Misunderstanding. Unless you keep at least frequently or LRU data local (along your Drupal boxes) all your nice global reachability concept will break at that point. Even just reading a DB is among the slowest, if not the slowest part.
Look: If I, a client in Europe, surf to one of your (dynamic) sites I end up at a cache box in say, NL. To serve, however, a connection must be made to a Drupal box, which again needs to connect to a DB. The evil genie in that bottle is that static sites are fast anyway; the beasts are the dynamic ones and if you want the global reachability thing working you'll need to a) include all parts and b) know where your bottlenecks and time consumers are. Getting that wrong might lead to basically just wastin money for global boxes.
I have worked on projects like that (and one was brutally massive on top). You must, in a way, build a "bucket chain", i.e. each layer must cache so as to avoid to needing to contact another layer (which is cached again).
I don't know how massive and complex your sites are but I'd tend to say that Drupal (and similar) won't cut it for anything but quite simple sites.
Not when reading from 5 Galera masters, it's not (especially with most of data served from DB cache in RAM).
90% of content will be served by the Varnish frontend with no need to query the backend. This said, Drupal backends will be replicated as neccessary (on the fly). Ditto re Varnish frontends.
Actually, I have found nothing out there that can match Drupal. Not to mention the automation possible with Drush. I've been friends with Drupal for 5 years, running sites 100x more complex than the ones coming. No worries here.
Oh, I didn't mean that Drupal is in any way bad. I was rather thinking in terms of a real server rather than some PHP/Python/Ruby engine with scripts.
My fault. Apologies.
Interesting, what do you mean by "real server"?
I'm sorry, I can't tell much about it (NDA), but I was working on a project where hundreds and hundreds of thousands of requests/s came in. Besides playing router tricks (of course) we also build a custom application server (compiled) with different (incl. DB) layers. To do that we basically needed a local DB manager and cache built in that would interact with the DB back end which again had its own front end (among other reasons to distribute workload over multiple DB servers.
But then, that was a major project with a pretty hefty dynamic nature. If your about 1000 domains are normal sites with average/typical low dynamics, Drupal or sth. similar might actually be a fine choice.
@bsdguy: actually, initially even Drupal is overkill here. Even Ghost would do. However, I decided on Drupal given the anticipated growth and expansian plus automation (scripting) needs.
There might be no reason for Anycast DNS as your main goal seems to be to serve the web page through the closest web server. You could use BIND with GeoIP to serve the closest web server:
http://blog.unixy.net/2010/07/how-to-build-your-own-cdn-using-bind-geoip-nginx-and-varnish/
http://geoipdns.org/ (djbdns)
https://code.google.com/p/geoip-tinydns/wiki/Overview
You can get BuyVM anycast and do internal routing using ExaBGP. With this you don't need anycast DNS but you will be limited by number of their PoPs
@tomle: yes, this is a viable option, thanks. While I am familiar with GeoDNS using BIND with Views, the djbdns option is new to me. I freely admit my research into this option has not been thorough enough due to my fascination with anycast DNS.
I subscribe to the KISS principle (Keep It Simple Stupid
Hence my preference for anycast DNS, stemming from my understanding that an anycast DNS server is a much more simple and lightweight solution than any GeoIP DNS option, where each DNS server needs to evaluate each DNS query against its GeoIP database, which in turn needs to be frequently updated on all DNS nodes. It is my understanding that anycast IP routing takes care of this dynamically, not to mention that it is much more reliable than MaxMind's GeoIP data, I'm told. This said, GeoDNS is a viable option, of course, one I need to explore further, especially with respect to my chosen DNS platform - NSD4 - which looks to be, by far, the fastest/most lightweight authoritative DNS around. Current project aside, I have other uses down the road for my DNS solution, "tested" on the 1000+ domains 
Hence DNS node performance is a concern, albeit not so much for the initial 1000+ domains subset, of course.
Interesting, but once anycast IP routing is in place, what is the advantage of ExaBGP routing vis a vis anycast DNS?
Actually, the more I think about it, BuyVM's existing 3 POPs seem like a good enough fit to start the ball rolling. Although I will need at least one more POP in Asia (Singapore) to satisfy my production needs, I think...
@aglodek
Sure, but anycast DNS will not solve your problem of routing to the nearest web-server. Anycast DNS is only answering the query from the client by the DNS server that is closest to the client. What IP it provides is a different story.
If you want to do anycast IP routing also for the web traffic, that will require anycast IP addresses. In that case BuyVM anycast is a good solution but only three PoPs. Your own real anycast will cost some money, you need your own /24 and ASN and upstream BGP.
Using the GeoIP solution you won't need anycast, the DNS server will just reply to the client with the IP adderss of the server that is closest to the client.
No, it'll respond based on the IP of the nameserver that contacted it for a lookup.
8.8.x.x and 4.2.x.x type nameservers which are anycasted and simply massive in size cause real issues for GeoIP DB's since they'll usually just point back to the states or what have you.
The nameservers is a good solution but I really think the OP should also have the LB's feeding off the anycast setup as well just so it's serving the data as well from the closet spot.
I mean, great, you got your record resolved in 20ms but it's shooting you to a pop in EU to serve, it's not doing you any good.
An Asia based pop is on our road map, it's just not very high on our TODO list yet as we just got Europe online, & we got many other things being worked on.
Thanks for the interest,
Francisco
Not true, most of modern public DNS servers support ENDS0 with client subnet option and GeoDNS is pretty accurate
http://vincent.bernat.im/en/blog/2013-exabgp-highavailability.html
Is rage4 still caching edns0 answers, ignoring subnet until the cache expires?
No, we do not cache EDNS0 enabled packages
For a record with 2 geodns ips, US/DE coordinates:
dig testedns.com A @ns1.r4ns.com +client=4.4.0.1Returns the US ip, which is correct. However, further requests (coming from the same DNS server) will ignore the subnet and return the US ip, no matter what. This seems to be cached for for ~60s.
What am I doing wrong?
@vld - read the docs regarding GeoDNS
the TTL should be 1 sec. and not 1 hour. Basically we have two types of cache enabled - one on packet level and one on backend level. The packets with EDNS0 are not cached (which is correct) but all backend results with TTL higher than 30 sec. are cached for performance reasons.
@aglodek - drop me a mail at office [ at ] gbshouse.com and we will craft something nice for you. For sure it will be cheaper than your own anycast infra.