New on LowEndTalk? Please Register and read our Community Rules.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.
Anyone setup SSD RAID 6?
randvegeta
Member, Host Rep
in General
Has anyone setup a RAID 6 array, only using SSDs?
How is the performance compared to both RAID 10 and single SSDs?
Any drive failures?
Comments
You'll kill drives in that configuration quicker than you would in RAID10 due to the extra writes. Performance should be lower than a RAID10 config with the same number of drives.
Tl;dr : not worth it with ssd
Are the number of additional writes really so much larger? I've read this before, and of course, given the parity drives, there is more work (writes) required to maintain proper parity, but is the additional write operations really so huge that there is a significant difference in life span?
As for performance, yes obviously RAID 10 would be faster, in the same way RAID 10 makes HDDs faster. But I would imagine that RAID 6 SSDs would have a significant performance boost over RAID 6 HDDs, and possibly still better than single disk performance.
I would setup a test machine, but I don't have many SSDs spare, so was wondering if anyone had any real word experience with an SSD RAID 6 array. The consensus definitly seems to be that RAID 6 will have worse performance than RAID 10 and drives will die sooner, but I have not been able to find any actual metrics for this.
Maybe the performance hit is not so large? Maybe the extra disk space makes up for the higher failure rate? Or maybe it's a complete waste of time because maybe RAID 10 HDD performance is comparable, but a lot cheaper.
or use SLC ssd as caching layer over large HDD arrays.
This is what we do already. So the real question is if the SSD cached HDD RAID 10 arrays have better performance than RAID 6 SSDs.
So far, SSD caching seems to have a nice cost/performance ratio, which is feasible to run even at the low end.
But RAID 10 SSDs means only 50% of your disk space is actually usable, and our current Virtuozzo Cloud Storage means only 33%, making SSD storage prohibitively expensive.
If the performance of RAID 6 SSDs is not significantly better than RAID 10 HDDs + SSD cacheing, then it's just not worth while. I suspect this is the case, but I've not found any data on this online. And I don't really fancy spending $2,000 just to test this.
>
Hello
I've got a server with 8x500 GB Samsung 850 Evo's running Raid 6 on HP Perc Raid card.
proxmox installed on it with some light usage linux vm's for a corporate customer.
Tell me what kind of test's you want to see, I'll do it for you.
9.75k IOPS? That's pretty impressive actually.
I assume the RAID card has caching in itself. Does it also have a battery?
Did you by any chance test the same physical setup on RAID 10?
LSI, for example, recommends to disable caching for SSD drives for better performance.
LSI also disable caching by default as soon as the controller recognize ssds while creating a new array.
If you don't use shitty drives, it doesn't really matter.
Most people won't exceed the endurance rating on enterprise disks anyway, if a disk actually dies before the endurance is up, it's easy to RMA it and get a new free disk.
For OP:
Yes, I've run with raid 6 with SSDs in about 600 servers using SM863 960GB disks - each array would consist of 22 drives in total, 2 of them being hot spares and the remaining 20 would be the actual raid.
It gave a usable space of roughly 16 terabytes (actual usable), where a similar raid 10 would give 8.94 terabytes.
This specific setup required a whole lot of storage but still enough parity for two disk failures, thus we went for raid 6. Rebuild times being like 35 minutes or so if I remember correctly.
We never had disks that died due to endurance being reached (it's about 6 petabyte on that specific version), disks would die for other reasons (such as power failure in a DC twice in one day, which took some equipment with it).
The performance was nice, reliability was nice - really nothing to complain about :-) and Samsung is amazing at handling RMA's for partners so ya.
Not sure I'd do it with consumer grade drives with a low endurance :-)
The workload by the way, was heavy random IO (reads and writes), with millions and millions of small files and sometimes big chunks of data as well.
Generally saturated the network before saturating the array.
Also @randvegeta - 9.75k iops test is sequential read in the test - so not really sure how impressive it is for an SSD array :-D
Ah. Missed the sequential part. Indeed nothing impressive for that then.
Hi,
Testnode: Dual Xeon, 384GB Ram, LSI SAS Controller, 12x 1TB SATA SSD Raid6 (WT, Direct IO, No Readahead)
I ran the tests as reported here: https://serverscope.io/trials/rmyz#io
dd:
FIO random read:
benchmark: (groupid=0, jobs=8): err= 0: pid=19313: Tue Jun 5 11:22:11 2018 read : io=810977MB, bw=13516MB/s, iops=3460.2K, runt= 60001msec clat (usec): min=0, max=7897, avg= 1.56, stdev= 6.49 lat (usec): min=0, max=7897, avg= 1.59, stdev= 6.49 clat percentiles (usec): | 1.00th=[ 1], 5.00th=[ 1], 10.00th=[ 1], 20.00th=[ 1], | 30.00th=[ 1], 40.00th=[ 1], 50.00th=[ 1], 60.00th=[ 1], | 70.00th=[ 1], 80.00th=[ 2], 90.00th=[ 2], 95.00th=[ 2], | 99.00th=[ 2], 99.50th=[ 2], 99.90th=[ 123], 99.95th=[ 129], | 99.99th=[ 141] bw (MB /s): min= 29, max= 2105, per=12.48%, avg=1687.31, stdev=666.97 lat (usec) : 2=76.25%, 4=23.33%, 10=0.03%, 20=0.14%, 50=0.01% lat (usec) : 100=0.01%, 250=0.25%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 10=0.01% cpu : usr=29.97%, sys=58.23%, ctx=524548, majf=0, minf=1053 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=207610010/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: io=810977MB, aggrb=13516MB/s, minb=13516MB/s, maxb=13516MB/s, mint=60001msec, maxt=60001msec Disk stats (read/write): sda: ios=524288/1, merge=0/0, ticks=59244/0, in_queue=59384, util=14.03%FIO random direct read:
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32 ... fio-2.2.10 Starting 8 processes Jobs: 8 (f=8): [r(8)] [100.0% done] [245.1MB/0KB/0KB /s] [62.1K/0/0 iops] [eta 00m:00s] benchmark: (groupid=0, jobs=8): err= 0: pid=19363: Tue Jun 5 11:23:49 2018 read : io=13314MB, bw=227223KB/s, iops=56805, runt= 60001msec clat (usec): min=105, max=246830, avg=138.39, stdev=1410.51 lat (usec): min=105, max=246830, avg=138.52, stdev=1410.51 clat percentiles (usec): | 1.00th=[ 111], 5.00th=[ 113], 10.00th=[ 115], 20.00th=[ 118], | 30.00th=[ 120], 40.00th=[ 122], 50.00th=[ 124], 60.00th=[ 126], | 70.00th=[ 129], 80.00th=[ 133], 90.00th=[ 137], 95.00th=[ 143], | 99.00th=[ 153], 99.50th=[ 161], 99.90th=[ 187], 99.95th=[ 354], | 99.99th=[ 7072] bw (KB /s): min= 5250, max=32840, per=12.59%, avg=28603.21, stdev=5197.84 lat (usec) : 250=99.93%, 500=0.03%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%, 100=0.01%, 250=0.01% cpu : usr=2.49%, sys=10.20%, ctx=3408462, majf=0, minf=359 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=3408396/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): READ: io=13314MB, aggrb=227222KB/s, minb=227222KB/s, maxb=227222KB/s, mint=60001msec, maxt=60001msec Disk stats (read/write): sda: ios=3404788/0, merge=0/0, ticks=434688/0, in_queue=435480, util=100.00%FIO random write:
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=sync, iodepth=32 ... fio-2.2.10 Starting 8 processes Jobs: 8 (f=8): [w(8)] [100.0% done] [0KB/1626MB/0KB /s] [0/416K/0 iops] [eta 00m:00s] benchmark: (groupid=0, jobs=8): err= 0: pid=19414: Tue Jun 5 11:25:17 2018 write: io=99063MB, bw=1651.3MB/s, iops=422663, runt= 60001msec clat (usec): min=1, max=533209, avg=18.04, stdev=303.53 lat (usec): min=1, max=533209, avg=18.10, stdev=303.53 clat percentiles (usec): | 1.00th=[ 3], 5.00th=[ 13], 10.00th=[ 14], 20.00th=[ 16], | 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 18], 60.00th=[ 18], | 70.00th=[ 19], 80.00th=[ 19], 90.00th=[ 20], 95.00th=[ 21], | 99.00th=[ 25], 99.50th=[ 30], 99.90th=[ 37], 99.95th=[ 39], | 99.99th=[ 229] bw (KB /s): min=63730, max=262896, per=12.53%, avg=211915.02, stdev=21246.01 lat (usec) : 2=0.02%, 4=1.12%, 10=0.82%, 20=80.15%, 50=17.87% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 750=0.01% cpu : usr=4.28%, sys=93.75%, ctx=257820, majf=0, minf=1317 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=25360247/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: io=99063MB, aggrb=1651.3MB/s, minb=1651.3MB/s, maxb=1651.3MB/s, mint=60001msec, maxt=60001msec Disk stats (read/write): sda: ios=0/32920, merge=0/13, ticks=0/907396, in_queue=907348, util=11.67%FIO random direct write:
Starting 8 processes Jobs: 8 (f=8): [w(8)] [100.0% done] [0KB/13706KB/0KB /s] [0/3426/0 iops] [eta 00m:00s] benchmark: (groupid=0, jobs=8): err= 0: pid=19465: Tue Jun 5 11:26:44 2018 write: io=784160KB, bw=13069KB/s, iops=3267, runt= 60003msec clat (usec): min=223, max=504109, avg=2435.67, stdev=21341.01 lat (usec): min=224, max=504110, avg=2436.32, stdev=21341.05 clat percentiles (usec): | 1.00th=[ 237], 5.00th=[ 239], 10.00th=[ 241], 20.00th=[ 245], | 30.00th=[ 245], 40.00th=[ 247], 50.00th=[ 249], 60.00th=[ 251], | 70.00th=[ 255], 80.00th=[ 262], 90.00th=[ 290], 95.00th=[ 868], | 99.00th=[84480], 99.50th=[171008], 99.90th=[321536], 99.95th=[378880], | 99.99th=[440320] bw (KB /s): min= 13, max=11217, per=12.88%, avg=1683.37, stdev=1728.26 lat (usec) : 250=52.56%, 500=39.34%, 750=2.56%, 1000=3.47% lat (msec) : 2=0.32%, 4=0.29%, 10=0.03%, 20=0.06%, 50=0.15% lat (msec) : 100=0.34%, 250=0.65%, 500=0.23%, 750=0.01% cpu : usr=0.39%, sys=2.32%, ctx=392138, majf=0, minf=176 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued : total=r=0/w=196040/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0 latency : target=0, window=0, percentile=100.00%, depth=32 Run status group 0 (all jobs): WRITE: io=784160KB, aggrb=13068KB/s, minb=13068KB/s, maxb=13068KB/s, mint=60003msec, maxt=60003msec Disk stats (read/write): sda: ios=0/195916, merge=0/11, ticks=0/53160, in_queue=53072, util=88.45%@randvegeta as expected direct random writes are a downside of raid6.
Is that software RAID or hardware RAID?
hardware Raid without special tuning/configuration of LSI controller. Just changed to WT, direct IO and readaheadnone.
I agree 100%. Wear is almost doubled. Avoid booth Raid 5/50 and Raid 6/60 with SSDs.
Also even big arrays of Raid 10 should be mix of drives. For example A row Samsung, B row Intel