How to test Storage/Disk I/O Performance in Linux machines

The most common performance bottlenecks you see as a programmer, administrator, or computer user is inadequate storage I/O performance. There can be many reasons for poor performance, but measuring it and understanding what needs to be measured and monitored is always a useful exercise.

In this article, we are going to see how to test the disk I/O performance in Linux machines with fio. We will show testing results for the following scenario :

• Performing a Random Write Test
• Performing a Random Read Test

Tools to check the I/O performance

In this section, we will discuss the different tools used for I/O performance testing, but our focus will be on fio The flexible I/O tester, this tool determines real-world performance.

fio – The flexible I/O tester

fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user. The typical use of fio is to write a job file matching the I/O load one wants to simulate.

Installation

fio is available in most of the distribution. The following command can be used to install fio:

Install fio in Ubuntu:

sudo apt install fio

Install fio in CentOS:

sudo yum install fio

fio commands parameters

fio has a lot of parameters. Most important are the following:

• --name: This assigned to the job. Also, this parameter has the special purpose of signaling the start of a new job.
• --filename:  It specify the filename to write from.
• --direct: Use bool value, if value is true, use non-buffered I/O. Default: false.
• --rw: Type of I/O pattern. For eg. if you want to  read (--rw=read) or write (--rw=write).
• --size:  The total size of file I/O for the test.
  • --io_size: Specifies how much I/O fio will do.
  • –filesize: To set individual file sizes.
• --blocksize:  It refers to block size in bytes used for I/O units. Default: 4096.
• --ioengine (I/engine):  Defines I/O test method to perform. Some options are following:
  • sync Basic read or write I/O. lseek is used to position the I/O location. See fsync and fdatasync for syncing write I/Os.
  • psync Basic pread or pwrite I/O. Default on all supported operating systems except for Windows.
  • vsync Basic readv or writev I/O. Will emulate queuing by coalescing adjacent I/Os into a single submission.
  • mmap File is memory mapped with mmap and data copied to/from using memcpy.
• --numjobs:  Mentioning the number of jobs more than one will perform the testing in parallel.
• --runtime: Setting the max run time, it makes fio terminate after a given amount of time.
• –group_reporting: If set, display per-group reports instead of per-job when numjobs is specified

There are a vast amount of parameters and it is not feasible to mention all of them here for reference see the manpage.

In the next section, we will see how to perform I/O testing.

Performing a Random Write Test

To perform a random write test use the following command. In the previous subsection, we have described different parameters.

In a nutshell, this command performs a random write which performs asynchronous I/O operations. We are using block size of 4k for a file size of 5g and the total number of threads will be 4 that will run at max 240 sec(4 minutes).

sudo fio --name=randomwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=5G --numjobs=4 --runtime=240 --group_reporting

Output of the above command mentioned below:

foofunc@ubuntu:~$ sudo fio --name=randomwrite --ioengine=libaio --iodepth=1 --rw=randwrite --bs=4k --direct=0 --size=5G --numjobs=4 --runtime=240 --group_reporting
randwrite: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.16
Starting 4 processes
randwrite: Laying out IO file (1 file / 5120MiB)
randwrite: Laying out IO file (1 file / 5120MiB)
randwrite: Laying out IO file (1 file / 5120MiB)
randwrite: Laying out IO file (1 file / 5120MiB)
Jobs: 2 (f=2): [_(2),w(2)][100.0%][w=566MiB/s][w=145k IOPS][eta 00m:00s]
randwrite: (groupid=0, jobs=4): err= 0: pid=38528: Wed Sep 22 22:14:53 2021
  write: IOPS=173k, BW=678MiB/s (711MB/s)(20.0GiB/30221msec); 0 zone resets
    slat (nsec): min=1406, max=51764k, avg=21831.84, stdev=726828.62
    clat (nsec): min=177, max=3181.1k, avg=390.14, stdev=2299.46
     lat (nsec): min=1655, max=51766k, avg=22329.90, stdev=726926.36
    clat percentiles (nsec):
     |  1.00th=[  199],  5.00th=[  215], 10.00th=[  223], 20.00th=[  235],
     | 30.00th=[  243], 40.00th=[  255], 50.00th=[  274], 60.00th=[  326],
     | 70.00th=[  386], 80.00th=[  446], 90.00th=[  644], 95.00th=[  924],
     | 99.00th=[ 1736], 99.50th=[ 1912], 99.90th=[ 2256], 99.95th=[ 2704],
     | 99.99th=[13248]
   bw (  KiB/s): min=533752, max=4372464, per=100.00%, avg=700678.83, stdev=125679.79, samples=238
   iops        : min=133438, max=1093116, avg=175169.65, stdev=31419.95, samples=238
  lat (nsec)   : 250=36.54%, 500=47.95%, 750=8.14%, 1000=3.23%
  lat (usec)   : 2=3.83%, 4=0.29%, 10=0.02%, 20=0.01%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%
  cpu          : usr=4.59%, sys=17.66%, ctx=3954, majf=0, minf=45
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,5242880,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=678MiB/s (711MB/s), 678MiB/s-678MiB/s (711MB/s-711MB/s), io=20.0GiB (21.5GB), run=30221-30221msec

Disk stats (read/write):
  nvme0n1: ios=4/3011884, merge=0/117094, ticks=13/12143203, in_queue=12143324, util=98.44%
foofunc@ubuntu:~$ 

Performing a Random Read Test

We will be running the command for the reading test with the same parameter which we used in the previous subsection.

sudo fio --name=randomread --ioengine=libaio --iodepth=1 --rw=randread --bs=4k --direct=0 --size=5G --numjobs=4 --runtime=240 --group_reporting

The output of the above command is mentioned below.

foofunc@ubuntu:~$ sudo fio --name=randomread --ioengine=libaio --iodepth=1 --rw=randread --bs=4k --direct=0 --size=5G --numjobs=4 --runtime=240 --group_reporting
[sudo] password for foofunc: 
randomread: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.16
Starting 4 processes
randomread: Laying out IO file (1 file / 5120MiB)
randomread: Laying out IO file (1 file / 5120MiB)
randomread: Laying out IO file (1 file / 5120MiB)
randomread: Laying out IO file (1 file / 5120MiB)
Jobs: 1 (f=1): [_(3),r(1)][100.0%][r=35.0MiB/s][r=8971 IOPS][eta 00m:00s]    
randomread: (groupid=0, jobs=4): err= 0: pid=39199: Wed Sep 22 22:32:56 2021
  read: IOPS=39.1k, BW=153MiB/s (160MB/s)(20.0GiB/133939msec)
    slat (usec): min=60, max=6831, avg=97.92, stdev=25.79
    clat (nsec): min=292, max=157611, avg=990.98, stdev=715.51
     lat (usec): min=60, max=6835, avg=99.23, stdev=26.07
    clat percentiles (nsec):
     |  1.00th=[  414],  5.00th=[  482], 10.00th=[  524], 20.00th=[  580],
     | 30.00th=[  644], 40.00th=[  740], 50.00th=[  836], 60.00th=[  924],
     | 70.00th=[ 1048], 80.00th=[ 1240], 90.00th=[ 1736], 95.00th=[ 2064],
     | 99.00th=[ 3024], 99.50th=[ 3536], 99.90th=[ 7328], 99.95th=[11712],
     | 99.99th=[20608]
   bw (  KiB/s): min=128136, max=170688, per=100.00%, avg=159104.10, stdev=2543.53, samples=1052
   iops        : min=32034, max=42672, avg=39775.93, stdev=635.89, samples=1052
  lat (nsec)   : 500=6.93%, 750=34.36%, 1000=25.20%
  lat (usec)   : 2=27.82%, 4=5.41%, 10=0.21%, 20=0.05%, 50=0.01%
  lat (usec)   : 100=0.01%, 250=0.01%
  cpu          : usr=3.33%, sys=13.42%, ctx=5243212, majf=0, minf=49
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=5242880,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=153MiB/s (160MB/s), 153MiB/s-153MiB/s (160MB/s-160MB/s), io=20.0GiB (21.5GB), run=133939-133939msec

Disk stats (read/write):
  nvme0n1: ios=5242787/920, merge=30/386, ticks=449172/258, in_queue=449541, util=100.00%
foofunc@ubuntu:~$ 

Other Tools

You can also use the following tools to check the performance of storage.

• hdparm: hdparm provides a command line interface to various kernel interfaces supported by the Linux SATA/PATA/SAS “libata” subsystem and the older IDE driver subsystem. For more information you can refer to hdparm manual page
• dd: dd package can be used to test the I/O performance of the disk by basically Copying the file. For more information refer dd command manpage

Summary

There are many tools that can be used to find out the I/O performance of the disk in Linux but my personal favorite tool is fio. In this article, we discussed how to install fio and how to use fio command to random read and random write.

If you face any problem or doubt while using above mentioned command. Don’t hesitate to comment, I am happy to answer your query.