Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!


HA on DO/Vultr via block move, floating IP - good or dumb idea?
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

HA on DO/Vultr via block move, floating IP - good or dumb idea?

raindog308raindog308 Administrator, Veteran

I've been playing with different HA setups that having systems in the same DC allows. For example pacemaker/corosync/DRBD with a floating IP.

However, I'm wondering if these are perhaps overly complicated for my modest needs. With DO or Vultr, I can use the API to move IPs and block storage. So what about something like this:

  1. node1 is primary with a DB, web, etc. running. node2 is standby. Both in same DC, can talk over private network.

  2. node2 every X seconds connects to the DB on node1 to see if it's working, makes a web call, etc. Maybe it also gets info from an external monitor.

  3. If something isn't right, it uses the DO or Vultr API to move the floating IP to itself and move the block storage to itself, then starts the DB, web server, etc.

That failover is automated. Once node1 is fixed, it could be changed to the failover node, so now it's monitoring node2. Or fail back. Of course, everything can be done with scripts.

This is very much how I've done failover in enterprise SAN setups for decades.

Granted, you may be ripping a mounted volume with an active DB writing to it, and you have to fsck, recover DB, etc. But you do avoid any risk of split brain (as you face with DRBD) and alleviate any need for fencing.

Thoughts?

Comments

  • jarjar Patron Provider, Top Host, Veteran

    Take it a layer higher. Floating IP for failover of two load balancers which each have 2 back ends.

    Thanked by 1FrankZ
  • raindog308raindog308 Administrator, Veteran

    jarland said: Take it a layer higher. Floating IP for failover of two load balancers which each have 2 back ends.

    But it's the back end that's the issue. How to have a database that's always available and always gives a consistent answer. For a highly available database where you need consistency, I can think of these configs:

    (1) some kind of super-expensive clustered filesystem SAN system (e.g. Oracle RAC) - not on my budget

    (2) database files on NFS or other network-shared system. Wouldn't choose it, but seen it done successfully but I don't have an HA NFS appliance anyway.

    (3) DRBD or other volume-level replication

    (4) Single master database-level replication, the replica becomes master in the event of a failover.

    (5) Multi-master replication - perhaps that's what you're thinking here? Supported by MySQL I think, not by Postgres yet.

    (6) A volume that can be moved in the event of failover, which is what I was describing.

  • jarjar Patron Provider, Top Host, Veteran

    @raindog308 said:

    jarland said: Take it a layer higher. Floating IP for failover of two load balancers which each have 2 back ends.

    But it's the back end that's the issue. How to have a database that's always available and always gives a consistent answer. For a highly available database where you need consistency, I can think of these configs:

    (1) some kind of super-expensive clustered filesystem SAN system (e.g. Oracle RAC) - not on my budget

    (2) database files on NFS or other network-shared system. Wouldn't choose it, but seen it done successfully but I don't have an HA NFS appliance anyway.

    (3) DRBD or other volume-level replication

    (4) Single master database-level replication, the replica becomes master in the event of a failover.

    (5) Multi-master replication - perhaps that's what you're thinking here? Supported by MySQL I think, not by Postgres yet.

    (6) A volume that can be moved in the event of failover, which is what I was describing.

    Yeah I'm all for the master-master replication of database.

  • vfusevfuse Member, Host Rep

    Your database should be able to do an election in case one of the db servers goes down. You would need at least 3 db servers with 1 primary.

    With most nosql solutions this works.

  • You know. Sometimes I wonder if you really a genius or just bored out of everything.

    Your idea solves a big issue that has been bothering me for last 3 weeks. Idk why I did not think of this. Thanks brother. > @raindog308 said:

    I've been playing with different HA setups that having systems in the same DC allows. For example pacemaker/corosync/DRBD with a floating IP.

    However, I'm wondering if these are perhaps overly complicated for my modest needs. With DO or Vultr, I can use the API to move IPs and block storage. So what about something like this:

    1. node1 is primary with a DB, web, etc. running. node2 is standby. Both in same DC, can talk over private network.

    2. node2 every X seconds connects to the DB on node1 to see if it's working, makes a web call, etc. Maybe it also gets info from an external monitor.

    3. If something isn't right, it uses the DO or Vultr API to move the floating IP to itself and move the block storage to itself, then starts the DB, web server, etc.

    That failover is automated. Once node1 is fixed, it could be changed to the failover node, so now it's monitoring node2. Or fail back. Of course, everything can be done with scripts.

    This is very much how I've done failover in enterprise SAN setups for decades.

    Granted, you may be ripping a mounted volume with an active DB writing to it, and you have to fsck, recover DB, etc. But you do avoid any risk of split brain (as you face with DRBD) and alleviate any need for fencing.

    Thoughts?

  • if you have files uploaded by your visitors, inotify+rsync
    web sessions store in the database
    multiple master replication

    The floating IP may become the new failure point so I would also add the possibility to switch to the real IP (dns update). For this case, no real need for a disk block.

  • For the database I would suggest MariaDB v10.1 which includes galera cluster. Very easy to set up a cluster and works great over slower WAN connections.

    You need 3 database servers at a minimum. So 2 sites + database only server will work. I have done a bunch of testing with it and it just works. Seems very robust.

  • raindog308raindog308 Administrator, Veteran

    seriesn said: You know. Sometimes I wonder if you really a genius or just bored out of everything.

    If I was better looking, I'd spend this energy having lots of sex. Instead, I'm stuck with computers.

    Doing this setup was nearly trivial. I want to productionalize it a bit but having a failover script running that checks the master and takes charge is very easy.

    A few tips:

    (1) I wanted something persistent and global so that when a node booted, it could look and see if it should be active or standby. etcd to the rescue. I love etcd. Thus, part of failover is to switch roles in etcd (e.g., "node1, you are now standby because I've taken over, so when you reboot, come back up as standby and start watching node2")

    (2) Hence, I disabled postgres, nginx, etc. in systemd startup and let my cluster daemon do it. It starts at boot and sees if it's supposed to be active, and if so, invokes the DO API to mount its storage and IP, starts Pg, etc.

    (3) To be neat and tidy, the failover script can reboot the failed node using the DO API. That's kind of slick. Most enterprise cluster solutions can't do that.

    I've been astonished at how fast block storage relocates in DO. It's a second or two, and IPs are virtually instant. Pg has done fine with crash recovery.

    hostingwizard_net said: if you have files uploaded by your visitors, inotify+rsync web sessions store in the database multiple master replication

    Assuming your database supports MM. Postgres doesn't yet. Also, you risk split-brain.

    hostingwizard_net said: The floating IP may become the new failure point so I would also add the possibility to switch to the real IP (dns update).

    DNS update is glacial. Floating IP is nearly instant. So...why DNS? Yes, it's a possible failure point but so is the rack I'm in, the switches that stuff flows through, etc. Either you trust the provider's infrastructure or you don't.

    moonmartin said: For the database I would suggest MariaDB v10.1 which includes galera cluster. Very easy to set up a cluster and works great over slower WAN connections.

    You need 3 database servers at a minimum. So 2 sites + database only server will work. I have done a bunch of testing with it and it just works. Seems very robust.

    I haven't used MariaDB but used Percona XtraDB Cluster and it worked great, even when I was doing DB servers on three continents. But...

    ...the more I developed my application, the more I kept running into MySQLisms that bug me...OK, that's a very old complaint and not something I just discovered. But no sequences, no schemas, no CTE...grrr...

    Anyway, not to make this a Pg vs. MySQL but if you want Pg, I think you need either (a) Pg replication + promote to master monitoring, or (b) moving storage. I chose (b) because it was much simpler.

    For MySQL, either native Multi-Master or one of the galera-based solutions. FYI, Percona states that 2-node setups work fine.

    Still experimenting with options...

  • You cannot use 2 databases for production galera cluster because you risk split brain. That is NOT a production solution.

    Native multimaster MySQL is a joke when you have a much more robust and simple solution like Galera as an option. It's no comparison.

    I have not done much testing with PgSQL multmaster but from what I have read about it, Galera is again, much more robust and simpler to use. It's a plug in replacement for MySQL v5.x but if you are not already using MySQL that is a different story.

  • raindog308raindog308 Administrator, Veteran

    moonmartin said: You cannot use 2 databases for production galera cluster because you risk split brain. That is NOT a production solution.

    So I'm aware of the odd-numbered clustering that's typical, but this is quoted straight from the current Percona XtraDB manual: "Recommended configuration is to have at least 3 nodes, but you can make it running with 2 nodes as well."

    Yes, I admit that having broken English in official docs is troubling...

    I hear what you're saying, though. Untangling databases where multiple nodes thought they were masters is...unpleasant.

    I'm skeptical that Percona has added any extra sauce...they probably mean as you suggest, "works fine 99% of the time...until you discover it didn't."

    moonmartin said: I have not done much testing with PgSQL multmaster but from what I have read about it, Galera is again, much more robust and simpler to use. It's a plug in replacement for MySQL v5.x but if you are not already using MySQL that is a different story.

    Historically, replication was not a focus of Pg but a few years ago they got wise. Multi-master isn't there yet, though there are third-party builds of Pg that support it. I'll wait for an official version.

    Pg does support aync or async to a RO replica, which you can even open for queries. But it's single-master and you have to promote on failover.

  • I am aware what Percona documentation says. The recommended minimum number for any (not just Galera) clustered database is 3 because of quorom.....full stop.

    Listen or don't. Test it or don't. All I can tell you is that I have searched high and low for a proper clustered solution for years. I have spent months trying out NoSQL. I have banged my head against the wall playing around with MySQL multimaster. I have tried re-learning databases by switching to PgSQL because there are PgSQL tribal factions who claim it's the best for everything and MySQL sucks.

    None of those solutions met expectations for a variety of reasons. Galera has. In fact it has blown me away just how well it works. I have databases on opposite sides of the earth that stay in sync now and it was surprisingly simple to do.

    Now that's it's built into MariaDB it's almost a no brainer. Only thing is you need to use InnoDB. Also the documentation is fairly decent as open source projects go.

  • raindog308raindog308 Administrator, Veteran

    moonmartin said: I have tried re-learning databases by switching to PgSQL because there are PgSQL tribal factions who claim it's the best for everything and MySQL sucks.

    Well MySQL does suck - it's clearly the worst among the major offerings (Oracle, DB/2, SQL Server, MySQL, Postgres)...no one can seriously contest that. However, that doesn't mean that it isn't the best choices in many cases. In many cases, because of ecosystem, MySQL is the right choice.

    Then again, in some cases, because of ecosystem, Windows is the right choice, so...

    moonmartin said: None of those solutions met expectations for a variety of reasons. Galera has. In fact it has blown me away just how well it works. I have databases on opposite sides of the earth that stay in sync now and it was surprisingly simple to do.

    Yes, MySQL seems to definitely have the lead in replication robustness (for FOSS - not in the same league as commercial offerings by a long stretch, alas).

Sign In or Register to comment.