All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Software RAID1 with SSDs instead of hardware RAID10 with mechanical drives?
With SSD prices coming down, does it make more sense to run software RAID with SSDs instead of paying for a hardware RAID controller and BBU, especially when considering the cost of SSD caching for the RAID controller?
Yes, there's quite a bit less space. Only one of our nodes is using over 1tb of storage, and most are under 500gb. A bit of data analysis earlier today shows that our average disk space allocation is 13 gigabytes, and average usage is 4.3 gigabytes.
I've typically been an ardent detractor of software RAID, but this is due to mdadm's psychotic self-checks that happen at the worst times and cause extreme load averages. This results from mdadm trying to check the disks while the processes running on the node are still trying to operate; I don't think this contention would result if using SSDs.
What are your thoughts? Would you use a host that's using software RAID with SSDs? Any other input on the subject?
Comments
We've had 70 nodes with SSDs and software RAID with no issue. Resyncing should only happen on a Sunday and even then it should throttle itself to not affect Disk IO
I agree it is worth a shot. My main gripe with mdadm are the annoying periods of time when the disk is being thrashed with a check/resync but also normal node activity (as you said).
So, maybe you should try SSD software RAID10, software RAID1, hardware RAID10, and hardware RAID1, and compare them!
@ShardHost Boss, what is the trick? Please teach me
.
@Damian For over 3 months now, tried with multiple SSD drives but LSI cachecade breaks and throws IO error in less than 3 hour after setting up fresh node. It cam to a point where I have decided to leave ssd and just use bigger raid arrays.
I'm considering going back to using software raid on future builds and we're having problems with controllers, SAS based.
Spent too much money this month just ordering new controllers, after failures.
We're going to a slightly bigger chassis now though, since the controllers are slightly above 1U right now. :-)
So we have a ghetto setup for our servers with lids balancing on top(sounds worse than it is), standard per U height is 1.75" and we're going 2.5" with customised chassis.
@Damian I believe it's a great idea, considering the fact that quality (good SSD controller is essential i think) SSDs are offering higher drive I/O speed thus lower CPU waiting time, compared to traditional average HDDs.
I was thinking about similar solution quite some time, and when the first opportunity arises i am going to realize it.
Have you tried Intel 520's? We have a Cachecade node with them and haven't had any issues, once we upgraded the controller's firmware.
Not yet. I will try em out next month. Lets see how bad this one breaks :P. Thanks
For 16-32 GB E3 we use this setup and it works well, never had an issue.
Touch wood...
It's a cron job and can be tweaked/disabled.
We use mdadm in systems that get absolutely hammered 24/7 right up to the IOPS limit the drives can support, as part of a large mail server farm. It has operated flawlessly for years. We have yet to experience a single failure attributed to mdadm itself (we've lost plenty of physical drives . . .)
Also damian, you're sacrificing the extra redundancy gain by using RAID 10, along with the storage vs dollar and then the big problem(for me) where you're going to be running nodes with SATA drives, and then nodes with SSD. While I have both SAS and SATA already, It's a pain keeping stock of different drives.
I'm fairly fond of the performance we're getting from 8x RE4 WD5003ABYX on LSI 9260-8i. It was actually at Gordon's suggestion over our planned 4x 1TB. Fairly heavy use just doesn't seem to have any real impact on IO. Dare I say it exceeds the 4x 1TB performance by more than double from what I'm seeing.
Granted I don't have the scale or years behind me that others do here.
Raid 10 usually scales in quite a linear fashion, so double the performance of your 4x1tb is expected.
I'd rather do RAID1, but 1tb SSDs are EXTREMELY expensive, $3k+.
I'd specifically like to stay with a 1u form factor... which is why i'm exploring improving disk i/o as much as possible. Disk i/o is currently the limiting factor, the E3's are able to keep up otherwise, and I don't think anyone ever runs out of memory doing OVZ....
You seem to doing a great job with catalyst host! I think transparency and honesty goes a long way.
@damian if you are willing to wait till Q2 check out the crucial m500 960 or 980gb (I forget which).
Sub $600.
Bigger Raid array, smaller storage = Guaranteed performance.
RAID 10 does not give you twice the redundancy of RAID 1, in the sense that most people think.
Statistically, our data has shown there to be little real redundancy gain between RAID 1 or 10. For us, it's a decision made purely on performance need. Of course, YMMV, depending on the type of drives used and environment operated in.
This is our reason for using RAID 10, too. More spindles = better iops.
sigh.... that is not how mdadm works at all, you need to read a book or 20, every thread you make yourself look worse.
I just ordered another server and was considering getting a pair of SSDs but for my customer base it's not worth it (yet). When I do get SSDs they'll be running in mdadm RAID1 though. No doubt about it.
@Damian
ahahahaha you win!
Aren't there cheaper alternatives for ssd cache raid controllers like the 9271 that include cachecade for around $400 total.
Also, I've been picking up brand new sealed 2TB RE4's shipped directly from WD from a wholesaler off eBay for $95 each.
As far as the 9211 go, I believe you can crossflash ibm m1015 into 9211-8i which can be bought for fairly cheap.
Just saw that. Thanks brother
No problem. I'm keen to try you in the future when im in need of another VPS
Oops, I guess I missed this in the first posting. If we move to using SSD drives, we'll be using only SSD drives; I don't want to deal with a SSD/SATA mix. The existing servers will be swapped out by attrition.
I'm seeing 9271-8i's for ~$670, but if they include cachecade, then there's that saved.
Be very careful with the SSD's you use. We are looking very hard at which SSD's we choose from now on because SSD's that do not have their DRAM cache's backed by batteries like the Intel DC S3700's (So far the only maker I've found that does this) you can wake up and find an entire array toast.
Intel 320's and Intel 520's do not have advanced power loss data protection. While it's fine in a laptop you cant yank the battery out of, it's a different matter entirely in a server if all power is lost. Probably the bigger reason Linode is holding off on migrating to SSD's because the price size ratio of SSD's with those battery backed DRAM caches are a lot more expensive. Though I dont see how a few capacitors can cost that much.
Samsung 830 had capacitors i think. But they are not sold anymore, unfortunately