Storage is one of the slowest components in a server's system, but is still the most important component. Storage is mainly used as a persistence mechanism to store a large amount of processed/unprocessed data. A slow storage device generally results in heavy utilization of read write buffers and higher memory consumption. You will see higher CPU usage, but most of the CPU time is spent waiting for I/O requests.
The recent developments of the flash storage medium have vastly improved storage performance. Still, it's one of the slowest performing components and needs proper planning— I/O planning in the application code, plus enough main memory for read write buffers.
In this recipe, we will learn to monitor storage performance. The main focus will be on local storage devices rather than network storage.
Getting ready
As always, you will need sudo access for some commands.
Some of the commands many not be available by default. Using them will prompt you if the command is not available, along with the process necessary to install the required package.
Install the sysstat package as follows. We have already used it in previous recipes:
$ sudo apt get install sysstat
How to do it…
The first command we will look at is vmstat. Using vmstat without any option displays an io column with two sub entries: bytes in (bi) and bytes out (bo). Bytes in represents the number of bytes read in per second from the disk and bytes out represents the bytes written to the disk:
Vmstat also provides two flags, -d and -D, to get disk statistics. Flag -d displays disk statistics and flag -D displays a summary view of disk activity:
There's one more option, -p, that displays partition-specific disk statistics. Use the command lsblk to get a list of available partitions and then use the vmstat -p partition:
Another command, dstat, is a nice replacement for vmstat, especially for disk statistics reporting. Use it with flag -d to get disk read writes per seconds. If you have multiple disks, you can use dstat to list their stats separately:
$ dstat -d -D total,sda
Next, we will look at the command iostat. When used without any options, this command displays basic CPU utilization, along with read write statistics for each storage device:
The column tps specifies the I/O requests sent to a device per second, and kb_read/s and kb_wrtn/s specifies per second blocks read and blocks written respectively. kb_read and kb_wrtn shows the total number of blocks read and written.
Some common options for iostat include –d, that displays disk only statistics, -g that displays statistics for a group of devices, flag -p to display partition specific stats, and -x to get extended statistics. Do not forget to check the manual entries for iostat to get more details.
You can also use the command iotop, which is very similar to the top command but it displays disk utilization and relevant processes.
The command lsof can display the list of all open files and respective processes using that file. Use lsof with the process name to get files opened by that process:
$ lsof -c sshd
To get a list of files opened by a specific PID, use the following command: $ lsof -p 1134. Or, to get a list of files opened by a specific user, use the $ lsof -u ubuntu command.
All these commands provide details on the read write performance of a storage device. Another important detail to know is the availability of free space. To get details of space utilization, you can use command df -h. This will list a partition-level summary of disk space utilization:
Finally, you can use the sar command to track disk performance over a period of time. To get real-time disk activity, use sar with the -d option, as follows:
$ sar -d 1
Use flag -F to get details on file system utilization and flag -S to display swap utilization. You can also enable sar logging and then extract details from those logs. Check the previous recipes in this article for how to enable sar logging. Also check manual entries for sar to get details of various options.
