Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Sign In Register

Quick Links


Categories

In this Discussion

› › How to test NVMe performance under linux
New on LowEndTalk? Please Register and read our Community Rules.

All new Registrations are manually reviewed and approved, so a short delay after registration may occur before your account becomes active.

How to test NVMe performance under linux

v3ngv3ng Member, Patron Provider
in Help

Hi,

I recently got a new dedi with two samsung sm961 NVMe SSDs and I wated to test the write/ read performance.
I'm using a Software Raid 1.
According to benchmarks, this SSD should write with 2.7 GB/s and read with 1.7 GB/s.

I have tested the write performance with the following command, but I "only" got around 700MB/s

dd if=/dev/zero of=testfile1 bs=1G count=1 oflag=direct 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.48908 s, 721 MB/s

Did I do something wrong and is this even the correct way to detect the write performance?

Regards

Comments

  • AlyssaDAlyssaD Member
    edited July 2018

    https://www.cyberciti.biz/tips/how-fast-is-linux-sata-hard-disk.html

    https://www.shellhacks.com/disk-speed-test-read-write-hdd-ssd-perfomance-linux/

  • AnthonySmithAnthonySmith Member, Patron Provider

    1GB really is not big enough for NVMe

    I think you want something more like:

     dd if=/dev/md2 of=/dev/null bs=4096k iflag=direct
^C1871+0 records in
1870+0 records out
15502868480 bytes (16 GB) copied, 4.28163 s, 3.6 GB/s

    Break it after a few seconds.

  • v3ngv3ng Member, Patron Provider

    Thank you @AnthonySmith

    dd if=/dev/md2 of=/dev/null bs=4096k iflag=direct ^C5371+0 records in 5370+0 records out 22523412480 bytes (23 GB, 21 GiB) copied, 8.63151 s, 2.6 GB/s

    But if I write e.g. 40GB with my previous command, the speed is much lower.

    dd if=/dev/zero of=testfile1 bs=1G count=40 oflag=direct 40+0 records in 40+0 records out 42949672960 bytes (43 GB, 40 GiB) copied, 34.3405 s, 1.3 GB/s

    I still cant explain why the results vary so much.

  • jsgjsg Member, Resident Benchmarker

    @v3ng

    OS caches. Plus writing all zeroes will strongly bend the result and not show the true performance. Note that the dd test is only good for a first rudimentary impression.

    You could download my vpsbenchmark linked in my signature. Your situation is one of the reasons I wrote it. Read the short docu and you'll find that you can also run it as a disk test only.

    vpsbench does both sequential and random access tests each reading and writing performance. Due to the fine granularity of timing you get solid results even with a small sample (you don't want to waste lots of lifetime of your NVMe just for benchmarking).

    Usage is very simple. Just download the binary for your system (probably linux 64 bit), have a quick look at the docu and run it.

  • teamaccteamacc Member

    @v3ng said:
    Thank you @AnthonySmith

    dd if=/dev/md2 of=/dev/null bs=4096k iflag=direct ^C5371+0 records in 5370+0 records out 22523412480 bytes (23 GB, 21 GiB) copied, 8.63151 s, 2.6 GB/s

    But if I write e.g. 40GB with my previous command, the speed is much lower.

    dd if=/dev/zero of=testfile1 bs=1G count=40 oflag=direct 40+0 records in 40+0 records out 42949672960 bytes (43 GB, 40 GiB) copied, 34.3405 s, 1.3 GB/s

    I still cant explain why the results vary so much.

    You are aware that the first test is a read test, and the second is a write test right? Read tests are generally faster.

  • lionlion Member

    bs=1G ... uhm

  • v3ngv3ng Member, Patron Provider
    edited July 2018

    @teamacc said:

    @v3ng said:
    Thank you @AnthonySmith

    dd if=/dev/md2 of=/dev/null bs=4096k iflag=direct ^C5371+0 records in 5370+0 records out 22523412480 bytes (23 GB, 21 GiB) copied, 8.63151 s, 2.6 GB/s

    But if I write e.g. 40GB with my previous command, the speed is much lower.

    dd if=/dev/zero of=testfile1 bs=1G count=40 oflag=direct 40+0 records in 40+0 records out 42949672960 bytes (43 GB, 40 GiB) copied, 34.3405 s, 1.3 GB/s

    I still cant explain why the results vary so much.

    You are aware that the first test is a read test, and the second is a write test right? Read tests are generally faster.

    Ugh, of course..

    So which command should I use for write tests? There are again pretty large differences, probably because of the filesize.
    Which size is the most realistic one?

    sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.835828 s, 1.3 GB/s

    dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.4278 s, 752 MB/s

  • jackbjackb Member, Host Rep
    edited July 2018

    Realistic? 4k; but you'll only really be seeing your iops.

  • v3ngv3ng Member, Patron Provider

    @lion said:
    bs=1G ... uhm

    Is this value too high?

  • FalzoFalzo Member
    edited July 2018

    don't use dd at all. use fio and run tests for different usual blocksizes like 4k and 64k.

    PS: if that's with a new hetzner box from their no setup-fee sale, please post a full smartctl on those nvme thingies :-)

    Thanked by 2vimalware mfs
  • nhocconannhocconan Member

    @Falzo said:
    don't use dd at all. use fio and run tests for different usual blocksizes like 4k and 64k.

    PS: if that's with a new hetzner box from their no setup-fee sale, please post a full smartctl on those nvme thingies :-)

    As per your request:

    # smartctl -a /dev/nvme0
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-24-generic] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZVKW512HMJP-00000
Serial Number:                      S316NX0JB00614
Firmware Version:                   CXA7500Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 512,110,190,592 [512 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Utilization:            33,061,912,576 [33.0 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Jul 18 11:48:09 2018 CEST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL *Other*
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     73 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.80W       -        -    0  0  0  0        0       0
 1 +     5.60W       -        -    1  1  1  1        0       0
 2 +     4.40W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1500
 4 -   0.0050W       -        -    4  4  4  4     2200    6000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        33 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    1,554,896 [796 GB]
Data Units Written:                 82,985 [42.4 GB]
Host Read Commands:                 539,143,144
Host Write Commands:                19,957,830
Controller Busy Time:               676
Power Cycles:                       18
Power On Hours:                     406
Unsafe Shutdowns:                   4
Media and Data Integrity Errors:    0
Error Information Log Entries:      7
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               33 Celsius
Temperature Sensor 2:               41 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0          7     0  0x0011  0x4016  0x004            0     1     -
  1          6     0  0x0011  0x4016  0x004            0     1     -
  2          5     0  0x0011  0x4016  0x004            0     1     -
  3          4     0  0x0011  0x4016  0x004            0     1     -
  4          3     0  0x0011  0x4016  0x004            0     1     -
  5          2     0  0x0011  0x4016  0x004            0     1     -
  6          1     0  0x0011  0x4016  0x004            0     1     -

    FIO

    # fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=712MiB/s,w=239MiB/s][r=182k,w=61.1k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=32181: Wed Jul 18 11:46:16 2018
   read: IOPS=170k, BW=665MiB/s (697MB/s)(3070MiB/4616msec)
   bw (  KiB/s): min=520016, max=729848, per=99.91%, avg=680410.67, stdev=80614.52, samples=9
   iops        : min=130004, max=182462, avg=170102.67, stdev=20153.63, samples=9
  write: IOPS=56.9k, BW=222MiB/s (233MB/s)(1026MiB/4616msec)
   bw (  KiB/s): min=172784, max=246376, per=99.93%, avg=227454.22, stdev=27245.43, samples=9
   iops        : min=43196, max=61594, avg=56863.56, stdev=6811.36, samples=9
  cpu          : usr=23.71%, sys=75.30%, ctx=2643, majf=0, minf=7
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=665MiB/s (697MB/s), 665MiB/s-665MiB/s (697MB/s-697MB/s), io=3070MiB (3219MB), run=4616-4616msec
  WRITE: bw=222MiB/s (233MB/s), 222MiB/s-222MiB/s (233MB/s-233MB/s), io=1026MiB (1076MB), run=4616-4616msec

Disk stats (read/write):
    md2: ios=768723/256943, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=392960/262660, aggrmerge=0/0, aggrticks=61552/3238, aggrin_queue=60104, aggrutil=95.76%
  nvme0n1: ios=382246/262660, merge=0/0, ticks=60480/3168, in_queue=59044, util=95.76%
  nvme1n1: ios=403674/262660, merge=0/0, ticks=62624/3308, in_queue=61164, util=95.76%


# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randread
test: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=0): [f(1)][100.0%][r=1080MiB/s,w=0KiB/s][r=276k,w=0 IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=32194: Wed Jul 18 11:47:00 2018
   read: IOPS=270k, BW=1055MiB/s (1106MB/s)(4096MiB/3884msec)
   bw (  MiB/s): min=  825, max= 1114, per=99.73%, avg=1051.78, stdev=105.50, samples=7
   iops        : min=211200, max=285396, avg=269255.71, stdev=27009.13, samples=7
  cpu          : usr=22.87%, sys=77.08%, ctx=34, majf=0, minf=72
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=1048576,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=1055MiB/s (1106MB/s), 1055MiB/s-1055MiB/s (1106MB/s-1106MB/s), io=4096MiB (4295MB), run=3884-3884msec

Disk stats (read/write):
    md2: ios=984258/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=524288/0, aggrmerge=0/0, aggrticks=45286/0, aggrin_queue=41292, aggrutil=96.00%
  nvme0n1: ios=514666/0, merge=0/0, ticks=44448/0, in_queue=40520, util=96.00%
  nvme1n1: ios=533910/0, merge=0/0, ticks=46124/0, in_queue=42064, util=96.00%


# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=670MiB/s][r=0,w=171k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=32205: Wed Jul 18 11:47:17 2018
  write: IOPS=165k, BW=645MiB/s (676MB/s)(4096MiB/6349msec)
   bw (  KiB/s): min=280640, max=695328, per=99.66%, avg=658375.33, stdev=118994.37, samples=12
   iops        : min=70160, max=173832, avg=164593.83, stdev=29748.59, samples=12
  cpu          : usr=22.54%, sys=73.25%, ctx=239, majf=0, minf=8
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,1048576,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=645MiB/s (676MB/s), 645MiB/s-645MiB/s (676MB/s-676MB/s), io=4096MiB (4295MB), run=6349-6349msec

Disk stats (read/write):
    md2: ios=0/1013342, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/1048648, aggrmerge=0/2, aggrticks=0/14184, aggrin_queue=6838, aggrutil=82.35%
  nvme0n1: ios=0/1048648, merge=0/2, ticks=0/13524, in_queue=6668, util=79.96%
  nvme1n1: ios=0/1048648, merge=0/2, ticks=0/14844, in_queue=7008, util=82.35%
    Thanked by 1Falzo
  • FalzoFalzo Member

    @nhocconan said:

    hey, thanks a lot!!

    Thanked by 1nhocconan
  • nhocconannhocconan Member

    @Falzo said:

    @nhocconan said:

    hey, thanks a lot!!

    You're welcome. I think you might not resist when keeping reading these posts :D

    At least posting a test is the only useful thing that I can do with my idle server collection atm :-D

  • v3ngv3ng Member, Patron Provider
    edited July 2018

    @Falzo said:
    don't use dd at all. use fio and run tests for different usual blocksizes like 4k and 64k.

    PS: if that's with a new hetzner box from their no setup-fee sale, please post a full smartctl on those nvme thingies :-)

    Sure, here you go

    
Model Number:                       SAMSUNG MZVKW512HMJP-00000
Serial Number:                      S316NX0J622884
Firmware Version:                   CXA7500Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 512,110,190,592 [512 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          512,110,190,592 [512 GB]
Namespace 1 Utilization:            504,322,023,424 [504 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Wed Jul 18 21:20:53 2018 CEST
Firmware Updates (0x16):            3 Slots, no Reset required
Optional Admin Commands (0x0017):   Security Format Frmw_DL *Other*
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Warning  Comp. Temp. Threshold:     70 Celsius
Critical Comp. Temp. Threshold:     73 Celsius

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     6.80W       -        -    0  0  0  0        0       0
 1 +     5.60W       -        -    1  1  1  1        0       0
 2 +     4.40W       -        -    2  2  2  2        0       0
 3 -   0.0400W       -        -    3  3  3  3      210    1500
 4 -   0.0050W       -        -    4  4  4  4     2200    6000

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning:                   0x00
Temperature:                        37 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    1%
Data Units Read:                    41,014,726 [20.9 TB]
Data Units Written:                 42,313,937 [21.6 TB]
Host Read Commands:                 5,915,093,081
Host Write Commands:                193,803,705
Controller Busy Time:               3,090
Power Cycles:                       19
Power On Hours:                     5,106
Unsafe Shutdowns:                   6
Media and Data Integrity Errors:    0
Error Information Log Entries:      9
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               37 Celsius
Temperature Sensor 2:               63 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
Num   ErrCount  SQId   CmdId  Status  PELoc          LBA  NSID    VS
  0          9     0  0x0011  0x4016  0x004            0     1     -
  1          8     0  0x0011  0x4016  0x004            0     1     -
  2          7     0  0x0011  0x4016  0x004            0     1     -
  3          6     0  0x0011  0x4016  0x004            0     1     -
  4          5     0  0x0011  0x4016  0x004            0     1     -
  5          4     0  0x0011  0x4016  0x004            0     1     -
  6          3     0  0x0011  0x4016  0x004            0     1     -
  7          2     0  0x0011  0x4016  0x004            0     1     -
  8          1     0  0x0011  0x4016  0x004            0     1     -
    Thanked by 1Falzo
  • jazz1611jazz1611 Member
    edited July 2018

    Hetzner AX160-NVMe, CentOS 7.5, Both NVMe use same model SAMSUNG MZQLW960HMJP-00003

    Partition below:
Swap 4GB
/boot ext4 512MB
/ xfs all

    Detail DISK for who needed

    [root@CentOS-75-64-minimal ~]# smartctl -a /dev/nvme0n1
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-862.9.1.el7.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Number:                       SAMSUNG MZQLW960HMJP-00003
Serial Number:                      S35XNX0K500265
Firmware Version:                   CXV8501Q
PCI Vendor/Subsystem ID:            0x144d
IEEE OUI Identifier:                0x002538
Total NVM Capacity:                 960,197,124,096 [960 GB]
Unallocated NVM Capacity:           0
Controller ID:                      2
Number of Namespaces:               1
Namespace 1 Size/Capacity:          960,197,124,096 [960 GB]
Namespace 1 Utilization:            2,196,074,496 [2.19 GB]
Namespace 1 Formatted LBA Size:     512
Local Time is:                      Thu Jul 19 10:00:31 2018 CEST
Firmware Updates (0x17):            3 Slots, Slot 1 R/O, no Reset required
Optional Admin Commands (0x000e):   Format Frmw_DL NS_Mngmt
Optional NVM Commands (0x001f):     Comp Wr_Unc DS_Mngmt Wr_Zero Sav/Sel_Feat
Maximum Data Transfer Size:         64 Pages
Warning  Comp. Temp. Threshold:     84 Celsius
Critical Comp. Temp. Threshold:     85 Celsius
Namespace 1 Features (0x02):        NA_Fields

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     9.00W       -        -    0  0  0  0        5       5

Supported LBA Sizes (NSID 0x1)
Id Fmt  Data  Metadt  Rel_Perf
 0 +     512       0         0
 1 -    4096       0         0

=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

SMART/Health Information (NVMe Log 0x02, NSID 0x1)
Critical Warning:                   0x00
Temperature:                        46 Celsius
Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    0%
Data Units Read:                    8,974,622 [4.59 TB]
Data Units Written:                 704,093 [360 GB]
Host Read Commands:                 1,485,113,438
Host Write Commands:                578,414,978
Controller Busy Time:               417
Power Cycles:                       4
Power On Hours:                     26
Unsafe Shutdowns:                   1
Media and Data Integrity Errors:    0
Error Information Log Entries:      0
Warning  Comp. Temperature Time:    0
Critical Comp. Temperature Time:    0
Temperature Sensor 1:               46 Celsius

Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged
    Thanked by 1Falzo
  • jazz1611jazz1611 Member

    Test FIO Mixed Read/Write

    "fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75"
testio: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
testio: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=454MiB/s,w=150MiB/s][r=116k,w=38.4k IOPS][eta 00m:00s]
testio: (groupid=0, jobs=1): err= 0: pid=31312: Thu Jul 19 10:09:58 2018
   read: IOPS=123k, BW=480MiB/s (503MB/s)(3070MiB/6400msec)
   bw (  KiB/s): min=459030, max=521034, per=100.00%, avg=493153.33, stdev=27572.25, samples=12
   iops        : min=114757, max=130258, avg=123288.25, stdev=6893.24, samples=12
  write: IOPS=41.0k, BW=160MiB/s (168MB/s)(1026MiB/6400msec)
   bw (  KiB/s): min=153552, max=175166, per=100.00%, avg=164982.08, stdev=9651.57, samples=12
   iops        : min=38388, max=43791, avg=41245.42, stdev=2412.91, samples=12
  cpu          : usr=20.03%, sys=78.95%, ctx=4897, majf=0, minf=372
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=785920,262656,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=480MiB/s (503MB/s), 480MiB/s-480MiB/s (503MB/s-503MB/s), io=3070MiB (3219MB), run=6400-6400msec
  WRITE: bw=160MiB/s (168MB/s), 160MiB/s-160MiB/s (168MB/s-168MB/s), io=1026MiB (1076MB), run=6400-6400msec

Disk stats (read/write):
    md2: ios=766816/256341, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=392960/262663, aggrmerge=0/0, aggrticks=70486/3021, aggrin_queue=73459, aggrutil=97.94%
  nvme0n1: ios=401779/262663, merge=0/0, ticks=71822/3113, in_queue=74882, util=97.91%
  nvme1n1: ios=384141/262663, merge=0/0, ticks=69151/2930, in_queue=72037, util=97.94%

    Test FIO Read Only

    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randread
testio: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [r(1)][100.0%][r=917MiB/s,w=0KiB/s][r=235k,w=0 IOPS][eta 00m:00s]
testio: (groupid=0, jobs=1): err= 0: pid=31371: Thu Jul 19 10:16:51 2018
   read: IOPS=232k, BW=906MiB/s (950MB/s)(4096MiB/4520msec)
   bw (  KiB/s): min=922872, max=942372, per=100.00%, avg=928601.78, stdev=6371.12, samples=9
   iops        : min=230718, max=235593, avg=232150.33, stdev=1592.66, samples=9
  cpu          : usr=16.55%, sys=83.43%, ctx=460, majf=0, minf=326
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=1048576,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=906MiB/s (950MB/s), 906MiB/s-906MiB/s (950MB/s-950MB/s), io=4096MiB (4295MB), run=4520-4520msec

Disk stats (read/write):
    md2: ios=1020111/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=524288/1, aggrmerge=0/0, aggrticks=43503/0, aggrin_queue=43555, aggrutil=97.69%
  nvme0n1: ios=535197/1, merge=0/0, ticks=44562/0, in_queue=44585, util=97.58%
  nvme1n1: ios=513379/1, merge=0/0, ticks=42444/0, in_queue=42525, util=97.69%

    Test FIO Write Only

    fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=testio --filename=testio --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
testio: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.1
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=513MiB/s][r=0,w=131k IOPS][eta 00m:00s]
testio: (groupid=0, jobs=1): err= 0: pid=31424: Thu Jul 19 10:17:57 2018
  write: IOPS=131k, BW=511MiB/s (536MB/s)(4096MiB/8012msec)
   bw (  KiB/s): min=510952, max=527342, per=100.00%, avg=523800.13, stdev=5300.65, samples=16
   iops        : min=127738, max=131835, avg=130950.06, stdev=1325.22, samples=16
  cpu          : usr=25.48%, sys=74.02%, ctx=1020, majf=0, minf=377
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwt: total=0,1048576,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
  WRITE: bw=511MiB/s (536MB/s), 511MiB/s-511MiB/s (536MB/s-536MB/s), io=4096MiB (4295MB), run=8012-8012msec

Disk stats (read/write):
    md2: ios=0/1033492, merge=0/0, ticks=0/0, in_queue=0, util=0.00%, aggrios=0/1048643, aggrmerge=0/0, aggrticks=0/13131, aggrin_queue=13059, aggrutil=97.97%
  nvme0n1: ios=0/1048643, merge=0/0, ticks=0/13217, in_queue=13124, util=97.87%
  nvme1n1: ios=0/1048643, merge=0/0, ticks=0/13046, in_queue=12994, util=97.97%
    Thanked by 1Falzo
  • jovpovjovpov Member

    @jazz1611 said:
    Hetzner AX160-NVMe, CentOS 7.5, Both NVMe use same model SAMSUNG MZQLW960HMJP-00003

    What tool/command did you use to make this hardware overview + disk & bandwidth benchmarks?

  • jazz1611jazz1611 Member

    @jovpov said:

    @jazz1611 said:
    Hetzner AX160-NVMe, CentOS 7.5, Both NVMe use same model SAMSUNG MZQLW960HMJP-00003

    What tool/command did you use to make this hardware overview + disk & bandwidth benchmarks?

    wget -qO- bench.sh | bash

  • jovpovjovpov Member

    do you have an URL for bench.sh? it looks really nice :)

  • mfsmfs Banned, Member
    edited September 2018

    jovpov said: URL for bench.sh

    Guess what, it's https://bench.sh/ ; I'd be wary of piping something right into bash without further inspection as general practice, but to each their own

    EDIT: btw I'd be wary of those I/O tests (dd based) as well, I'd second fio as per Falzo's suggestion

  • jsgjsg Member, Resident Benchmarker

    I recommend my Vpsbench tool (see my sig). Not because I wrote it but because it DOES cope with NVMe and reports sensible results. I know because I DID test NVMe VPS and dedis and it worked fine.

    Note: Normally Vpsbench offers an all in one (like @jovpov's screen) but using a command line switch one also can test/benchmark only a disk. It's documented in a short but completed manual.

    Thanked by 1yaxewenaj
  • IonSwitch_StanIonSwitch_Stan Member, Host Rep

    Thermal throttling is a huge concern for NVME disks, and can result in 50%+ performance throttle during long writes.

Sign In or Register to comment.