Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


highly available rest service on lowend vps
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

highly available rest service on lowend vps

AdamMAdamM Member
edited July 2015 in General

Hi,

I would like to create a highly available rest service with a set of lowend vps's. The service is simple, does a database call and returns the result.

By highly available, I mean that if one app server is down (crashed) or not accessible due to network problems (or the vps itself is down), then another app server should take over in its place. Note, I am not asking about making the database highly available, just the app servers themselves. Any suggestions on how to do this? Here is what I am currently thinking:

[1] I could use a single VPS as a reverse proxy, like HAProxy, in front of two or three app servers. If one of the app servers goes down, the proxy will fail-over to one of the backup app servers. This should work, but it seems to me that the VPS that runs the proxy is just as likely to fail as the actual app servers themselves. So, really this options actually adds an additional point of failure.

[2] Use a load-balancing service that has fail-overs, like Linode's nodebalancers. But, this is getting a bit more expensive. AWS also has a similar offering. And it may reduce to option #1 above, just managed by people who are smarter than me.

[3]... any other ideas?

Thanks,
-Adam

Comments

  • I looked into offering a service very similar to what you're describing the need for (and still may do it). Basically there would be 2 load balancers pointing to 2 (or more) vps's in the back end. It takes a bit of work because you have to think about what type of calamity you're protecting against.

    What I mean by that - is it just the VPS you're concerned with or is it the whole provider's network you're concerned about being unavailable? How far out into the Internet are you worried about the outage? if you would want to go to a single provider on, say, two hostnodes - that is very attainable and not too expensive (basically it would take 4 VPS's) but that does nothing if the provider has a network outage.

    To set it up for yourself isn't hard it's just that your load balanced nodes might be exposed to the Internet. Depends, again, what you're trying to protect against.

    This isn't my best work as an explanation but I guess I'm looking for some more information from you to give you some guidance. You could do this on LEB.

  • hbjlee17hbjlee17 Member, Host Rep
    edited July 2015

    2 reverse proxies with dns round-robin in different datacenters (preferably with low latency between the two datacenters), can't be highly available when your reverseproxy is the single point of failure, so only 1 reverseproxy won't do.

    4+ application server ( 2 in each datacenter )

    2+ database servers(if databases are needed) in either master master replication or other form of replication methods (one master in each datacenter)

    We, Lunanode, can provide this but the latency between our two datacenters is at around 20ms, if your application does not do latency sensitive transactions, then it should not be a problem.

    Thanked by 1bitseeker
  • AdamMAdamM Member

    @hbjlee17 , thanks, so if there are 2 reverse proxies with R-R DNS, what happens if one of the proxies goes down?

    Lets say we have proxy A and B.
    A typical web browser request is made, and the DNS resolves to proxy A,
    but proxy A just went down,
    does the web browser perform another DNS request, which then will get back proxy B? I am just not quite sure how this would work.

  • hbjlee17hbjlee17 Member, Host Rep
    edited July 2015

    When a DNS request is made, if DNS round-robin is setup properly it should return both your reverseproxy ip to the browser. you can test your dns setup by running this in linux console: dig @8.8.8.8 domain.com

    Most modern browsers are smart enough to handle the situation where one of the ip is unreachable that it will use the other ip instead.

    Thanked by 1bitseeker
  • AdamMAdamM Member

    hbjlee17, I think that fills in the gap for me.

  • ClouviderClouvider Member, Patron Provider

    You can also use DNS provider who monitors the state of application and will return only IPs of healthy nodes.

    Such services usually also do Geo IP balancing etc.

    Thanked by 1bitseeker
  • AdamMAdamM Member

    seems like DNS caching may make DNS failover services not fully HA... right?

  • hbjlee17hbjlee17 Member, Host Rep
    edited July 2015

    @AdamM said:
    seems like DNS caching may make DNS failover services not fully HA... right?

    Correct,

    Your best option would be to go with DNS round-robin as I explained above, I am unaware of other methods that offer close to immediate fail-over.

  • AdamMAdamM Member

    round robin seems easiest, but I seem to run into some info that says that browsers do not always follow the next ip... perhaps a little test is in order for my afternoon.

  • FranciscoFrancisco Top Host, Host Rep, Veteran

    How about an anycast HA cluster? :)

    http://buyvm.net/anycast-vps/

    You can 'float' an IP within each DC, so you could have 2 VM's on different nodes and if you need to do a maintenance or if there's an issue keepalived/etc can move the anycast IP around.

    Francisco

  • hbjlee17hbjlee17 Member, Host Rep
    edited July 2015

    @AdamM said:
    round robin seems easiest, but I seem to run into some info that says that browsers do not always follow the next ip... perhaps a little test is in order for my afternoon.

    I have tested this on my application and as previously mentioned, modern browsers have no problem using other IPs if one fails. But do test it : )

  • AdamMAdamM Member

    it seems like RR DNS is the best option for me. But, I am not looking for Load balancing but, fail-over. Are there any DNS providers that, when you specify more than one IP, it sends the IPs in the specified order, as opposed to the standard Round Robin random order? In this case, my main server would typically get the requests, and the backup would handle the request only when there is a necessary fail-over.

  • cassacassa Member

    Rage4 provides this

  • ClouviderClouvider Member, Patron Provider

    +1 Rage4
    @gbshouse

  • perennateperennate Member, Host Rep
    edited July 2015

    AdamM said: it seems like RR DNS is the best option for me. But, I am not looking for Load balancing but, fail-over. Are there any DNS providers that, when you specify more than one IP, it sends the IPs in the specified order, as opposed to the standard Round Robin random order? In this case, my main server would typically get the requests, and the backup would handle the request only when there is a necessary fail-over.

    If both IP addresses are provided in DNS, the browser is free to (and will) arbitrarily choose between them. There is no priority on A records like there are on MX records.

    You would need to either update the DNS record, in which case the failover time is going to be longer than when you have round-robin DNS setup, or forward requests from your backup to your primary.

  • hbjlee17hbjlee17 Member, Host Rep
    edited July 2015

    AdamM said: it seems like RR DNS is the best option for me. But, I am not looking for Load balancing but, fail-over. Are there any DNS providers that, when you specify more than one IP, it sends the IPs in the specified order, as opposed to the standard Round Robin random order? In this case, my main server would typically get the requests, and the backup would handle the request only when there is a necessary fail-over.

    Since the A records are going to be for the reverseproxies, you can optionally configure them to always talk with your master app server, and only communicate with the secondary app server when needed. The reverse proxies are really only there to ensure that when one DC is offline clients will still be able to reach your application server via the other reverseproxy.

    Ultimately, relying on dns server to determine whether to send your master app server IP or secondary app server IP in the dns response will not provide you with immediate fail-over due to:

    1. DNS caching at varies levels.

    2. It will most certainly take longer for the dns provider to determine whether your master app server is up than simply to use the reverseproxies to do the same.

Sign In or Register to comment.