New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
how to do server redundancy?
Already second time in little over a year dedicated server failed. There is apache2, mysql, postfix and courier-imap running.
What I would like is to keep auto-updated redundancy spare server in another location and automatically switch to it during failure and automatically sync data and switch back when main server comes back online.
How to synchronize the files between servers? How to setup DNS to auto-switch in case of failure?
Comments
Two good write-ups:
https://www.digitalocean.com/community/tutorials/how-to-configure-dns-round-robin-load-balancing-for-high-availability (including links to other articles for DB replication)
https://www.digitalocean.com/community/tutorials/how-to-create-a-redundant-storage-pool-using-glusterfs-on-ubuntu-servers
rSync & MySQL replication.
Hello,
I think you ask for config tips but as soon as there is a database it is more complicated it requires sthg like heartbeat to replicate datas between two servers.
Why dont you just ask for raid on your dedicated server ? it a disk fails it can be changed and if sthg more critical liek motherboard fails your provider should be able to takes the disks and place them in another server even if he also has to move the raid controler.
or you can use rsync and mysql replication like wych said easier maybe
As mentioned by Ikoula I would presume you mean the disk has failed? If it's happened more than once I'd be looking into RAID, it's much cheaper than buying a whole new server just to deal with one issue, especially if traffic is able to cope easily.
You could use rSync to achieve this though, goodluck
Whatever you'd do, try to keep it simple: no mysql master-master replication. I usually use rsync and never had a problem between VPS failures.
For the DNS auto-switch, you could declare the back-up server as a backup mx for the mail service. Regarding the A/AAAA resource, you could use UptimeRobot + API.
Whatever you'll do, please share the experience
RAID is excellent idea, although it only helps after single drive failure event.
Right now I am working on email redundancy.
The idea is to have MX 10 for mail.mydomain.com primary server and MX 20 for mail2.mydomain.com backup server. I am assuming email will only go to one of them, primary server will be always sent to first, if unavailable, sender will deliver to backup. Because we use IMAP with Maildir format, every email is a file, so frequent sync of /var/vmail folders between mail and mail2 servers should provide sufficient backup that could be used as temporary primary even for few days.
Both servers will also forward email to 3rd party webmail account which will automatically purge old emails, this is for temporary email access during the disaster event and also catch messages between last sync and disaster event.
Any thoughts ?
I have the DNS switching covered if you're using cloudflare with Uptime Robot or StatusCake.
You can also use DNSMadeEasy and their DNS failover service. I have used them for 5 years now, and my sites is "never" down, most 1-2 minutes, often less. The DNS fail over service kicks in pretty fast. And I have it only on medium, you can get it faster then that.
You can use cloudflare free DNS & update them when monitor detect an issue using their API ;-). Set either a low TTL for DNS or use them as reverse proxy.
Thanks guys. How about the email redundancy? See my longer post above.
Th> @HyperSpeed said:
The rsync part is easy but MySQL replication is a PiTA. You can't just set it and forget it either. You have to constantly keep an eye out for corruption and losing sync. Also the whole master-slave vs multi-master are both very different set ups. There is a free tool for MySQL clustering that makes it a bit easier. Can't remember the name. I've looked at doing this a few times. Did some tests and each time I concluded it was too much hassle.
Better to use NoSQL for clustering but most things people use for websites won't work with NoSQL.
Hello,
Concerning email redundancy i am not expert but i guess i would cron rsync command and create two MX records with different priority.
If you can manage a virtual ip you can make you webmail point at that Vip.
Tell me if you want me to rephrase that.
MariaD+Galera . or Percona XtraDB Cluster (galera based) both do multi-master synchronous WAN replication.
You might want to minimize latency between the masters with Galera. (affects max effective queries/s)
For a start don't use 20 as backup prio. According to the RFC even 11 would do, I know. But in the real world things are different and there are, for example, servers who try the backup MX if the primary MX isn't available right away unless the backup MX is evidently a backup. One algorithm seen in the wild out there is "if prio of secondary <= 4 times primary prio" then consider it acceptable to send right away. I've seen that game played even with "8 times".
Short version: A secondary MX should have a prio > 4 (or even 8) times that of the primary. So, better have MX 10 and MX 90.
Depending on your mail server(s) that might be a pretty reliable way to create confusion or even havoc.
There are well established (and working) mechanisms to handle that and you'd better use them.
Still shocked by the phpmysqladmin thread and quite some answers there I strongly advise against galera, percona and other "pre cooked and simple" replication solutions. You'll end up like the guy who is f-cked when his funny phpmysqladmin thingy or some funny DB panel plays tricks, gets confused or the like!
It might not please you but the sad truth is: Databases are by far more complex than phpmysqladmin might have you believe. With replication the complexity increases (losely speaking) to the square.
There's no way around it. If you want to do serious stuff you need serious understanding.
For a start it's a major difference whether redundancy is meant as resilience vs. as speed improvement or geo distribution.
Next for a reasonably designed and properly working setup there will pretty much always be a need to profoundly know and understand - and probably design or change - the application and all the gimmicks of your interpreters like code and data caching.
Yes, Cluster isn't beginning and end of redundancy.
I'd probably add a time delayed (regular type) mysql read-slave to any cluster.
Going pure-InnoDb lets you make mysql admin experience less shitty right?
Agreed.
Are you suggesting giving up ACID compliant RDBMS and going the 'noSql' route?
(I'm Not saying Mysql should be first choice for a greenfield oss project requiring RDBMS if starting today.)
No. I'm suggesting that sadly many questions here show the real problem that, however, isn't seen but trampled over with mysql dadaisms, panel clusterfucks and marketing shit bingo wisdom.
I'm suggesting that "democracy" is not a replacement for know-how and solid technology. Most importantly I'm suggesting that not-trivial applications are complex and additionally many components (like DB) are complex, to. While galera, phpmysqladmin, percona and whatnot seem they are the solutions they are in fact not. Not because they are shitty - some of them aren't - but because they can't replace a proper understanding of the scenario nor a solid level of know-how that is required.
And I'm suggesting, if I may shortly touch a political perspective, that we're deeply - and in the long run very painfully - f-cking ourselves.
In a sane world, non specialists understand their limits when they encounter high complexity. In our insane world, however, people who would have a hard time to properly set up their LAMP feel perfectly comfortable to (idiotically, pardon me) fumble around with scenarios that clearly (and for a sane mind obviously) require professional know-how. And that, in fact, starts with kust understanding the problem, let alone solving it.
But more often than not, professionals aren't asked or tasked. Instead people fall for the marketing and PR lies of the large business interests behind and aside linux, mysql, etc. and seriously believe clicking in sime miracle panel might actually work.
Unfortunately, all this mayhem doesn't come for free. It costs, among others, the demise of many professionals because without jobs they can't thrive. I'm not expecting many followers on my view, quite the contrary, I'm expecting harsh reactions, but I wanted to at least have mentioned it. What we do is slowly killing the very know-how that will be needed to escape that devils circle.
Oh and btw., why are there millions of customer records stolen almost weekly? Maybe, just maybe, because they were on systems and in databases that were setup by incapable admins who never understood the problems in the first place (and thought clicking in a panel would so the job ...)?
Is anybody doing it this way for websites?
Client from La Vegas gets geo-located to website in Los Angeles. Enters form data, form data is entered into DB on Los Angeles server as well as other servers in Montreal, France, and Hong Kong at the same time by the application that is accepting the form data on the server in Los Angeles. Done, all DBs in sync.
EDIT: Application reads from one, writes to all.
@Frankz:
Bear in mind the high latencies between those locations will affect the time-to-final-commit if using Galera+InnoDB.
On the plus side, if you have a ton of SELECT queries, then having a 'local' cluster node will really help; except that the qps for writes/updates that would be capped at a certain limit because of the synchronization and locking.
It really depends on what rows get locked by your queries' write pattern. Append-only would be better than read-update-row for instance.
Benchmark everything.
Plan for the entire cluster to fail catastrophically. Distributed anything is Hard. Edge cases galore.
(deleted)
EDIT: nm
@FrankZ#
(deleted)
P.S. How funny,
I'm writing my ass off to help you ... only to find that your post has been cut down to "nm".
I thought I had strayed too far off topic, and did not want to hijack the thread. It was removed within 10 minutes. More then happy to continue in PM