Newer Linux kernels have per-process I/O accounting and you can use the iotop tool to find out what’s performing I/O, but in many cases I’m trying to find the source of an I/O problem in an older kernel. I found sort of a hack-ish way to do that today, while trying to figure out why a system was basically unresponsive.
I found a post on Stack Overflow that showed a way you can get per process I/O statistics from the kernel even in older kernels. I adapted this to my needs, and wrote a little script.
Here’s how you use it. First, get it:
wget http://aspersa.googlecode.com/svn/trunk/iodump
Then turn on kernel messages about I/O:
echo 1 > /proc/sys/vm/block_dump
This makes the kernel start writing messages about every I/O operation that takes place. Now all you have to do is get those messages and feed them into my script:
while true; do sleep 1; dmesg -c; done | perl iodump
Wait a little while, then cancel the script. The results should look something like the following:
root@kanga:~# while true; do sleep 1; dmesg -c; done | perl iodump ^C# Caught SIGINT. TASK PID TOTAL READ WRITE DIRTY DEVICES firefox 4450 4538 251 4287 0 sda4, sda3 kjournald 2100 551 0 551 0 sda4 firefox 28452 185 185 0 0 sda4 kjournald 782 59 0 59 0 sda3 pdflush 31 30 0 30 0 sda4, sda3 syslogd 2485 2 0 2 0 sda3 firefox 28414 2 2 0 0 sda4, sda3 firefox 28413 1 1 0 0 sda4 firefox 28410 1 1 0 0 sda4 firefox 28307 1 1 0 0 sda4 firefox 28451 1 1 0 0 sda4
I deliberately generated a bunch of I/O by deleting my Firefox history and cache.
Got any better ideas, warnings, etc? Post them in the comments.
iotop (http://guichaz.free.fr/iotop/) is not too dissimilar from this, only giving real-time usage statistics, without the need to sample for a while before reviewing.
__
23 Aug 09 at 4:42 am
is there similar way to see I/O dump or mac osx? i use osx86 on my laptop and week ago , I badly needed to see I/O write activity which was in MB/s constantly with no real activity carried by me.
anshuman gholap.
anshuman
23 Aug 09 at 6:47 am
Good tip, thanks!
It’s minor, but I think “while true; do sleep 1″ can be replaced by “watch 1″
Ludwig
23 Aug 09 at 6:58 am
Another good tool is iotop. Debian users can simply “sudo aptitude install iotop”. iotop provides real-time disk i/o stats. Even without cumulative totals, it’s still fairly useful!
Sohail Mirza
23 Aug 09 at 7:02 am
sysstat 8.1.7-1 from ubuntu contains pidstat tool
It’s useful for per-proc io statistic with -d key
This is an example.
pidstat -d 1
Linux 2.6.31-rc5-generic (vega) 08/23/2009 _i686_ (1 CPU)
04:20:05 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
04:20:06 PM 6452 41009.71 0.00 0.00 dd
04:20:06 PM PID kB_rd/s kB_wr/s kB_ccwr/s Command
04:20:07 PM 6452 39920.79 0.00 0.00 dd
Nickolay Ihalainen
23 Aug 09 at 8:20 am
Right, pidstat, iotop, iopp — all are very useful, but not on an old CentOS server that doesn’t have per-process IO accounting :-)
Xaprb
23 Aug 09 at 1:41 pm
I/O accounting should be part of RHEL/CentOS 5.4.
http://dag.wieers.com/blog/red-hat-backported-io-accounting-to-rhel5
Bruce Locke
23 Aug 09 at 3:09 pm
Have a look at blktrace (the rpm or deb should include blktrace, blkparse, btrace and btt). Just point it at a block device and run for a while. Very useful. Lots of good howto’s turn up on google if you care to look.
John McNulty
23 Aug 09 at 6:16 pm
[...] How to find per-process I/O statistics on Linux at Xaprb (tags: linux sysadmin performance monitoring tools process kernel tips) [...]
links for 2009-08-23 at DeStructUred Blog
23 Aug 09 at 10:03 pm
Why not write a single script to do everything required, something like:
#!/bin/sh
dmesg -c
/etc/init.d/klogd stop
echo 1 > /proc/sys/vm/block_dump
# allow 30 seconds of stats to be logged
sleep 30
dmesg -c | perl iodump
echo 0 > /proc/sys/vm/block_dump
/etc/init.d/klogd start
Tom Collins
24 Aug 09 at 12:31 am
John, I’ll look into that, thanks for the tip.
Tom, I wrote it as a separate script because some systems actually don’t have a klogd running as a daemon (and/or you might not want to disable it), and I also want to be able to log on one system and analyze on another system.
Xaprb
24 Aug 09 at 8:27 am
[...] How to find per-process I/O statistics on Linux at Xaprb (tags: linux sysadmin performance io monitoring) [...]
andy.edmonds.be › links for 2009-08-25
25 Aug 09 at 8:35 pm
[...] Schwartz shared How to find per-process I/O statistics on Linux. He writes, “Newer Linux kernels have per-process I/O accounting and you can use the iotop [...]
Log Buffer #159: a Carnival of the Vanities for DBAs | Pythian Group Blog
28 Aug 09 at 1:05 pm
Could you include cpu percent, memory usage (which working set in windows XP) for all processess in this?
Also i would like to know about the unit of i/o information. Is it KB or Bytes? Thank you in advance.
Sanjeev
25 Jun 10 at 3:38 am
Last post got no answer, I’d very much like to know as well – what units is this measuring? B, kB, MB?
sahumani
19 Oct 10 at 4:21 am
I don’t remember. If you find out, post the answer in a new comment.
Xaprb
19 Oct 10 at 6:47 am
Looks like the unit is simply “times”, like when you ask How many times has this process done READ’s (or WRITE’s)? Every time, a READ (or WRITE) reads (writes) one block. Not sure how vm/block_dump records scatter/gather I/O (multiple blocks per read or write). I guess it breaks it up into multiple READ/WRITE lines but I’m not sure. The next question is How big is the block? I’m pretty sure it’s the file system block size as when you created the file system. With that info, you can convert the iodump result into KB or MB.
Yong Huang
17 Nov 10 at 11:59 am
Here is the script :
#!/bin/bash
dmesg -c
echo 1 > /proc/sys/vm/block_dump
sleep 30
go=0
while [ $go = 0 ]; do
sleep 1
clear
dmesg | perl iodump
trap “go=1″ SIGINT
done
echo 0 > /proc/sys/vm/block_dump
dmesg -c
clear
echo Caught SIGINT.
terrabit
18 Dec 10 at 3:41 am
The unit for block_dump data is blocks, and the easiest way to determine that size in a way that works for any filesystem type is:
/sbin/blockdev –getbsz /dev/sda1
Note that even though I/O accounting showed up in RHEL 5.4, you still can’t get iotop to work because of other dependencies. As of iotop 0.4, it will compile and run against the Python 2.4 included with RHEL, so long as you’ve installed the python-ctypes package. But iotop still doesn’t work. Even though the main per-process I/O patch was backported in RHEL 5.4, the current kernel isn’t configured correctly for it, and there are other issues with the backport being incomplete. If you build iotop and run it, against the most recent RHEL 5.5 kernel I have here (2.6.18-194.26) it gives the error “CONFIG_TASK_DELAY_ACCT not enabled in kernel, cannot determine SWAPIN and IO %” and the detail level is accordingly low.
There is an open enhancement request to sort this out so iotop works:https://bugzilla.redhat.com/show_bug.cgi?id=557062 but it keeps missing being included in RHEL releases. It was hoped for in 5.6 but missed that deadline. The kernel patches needed are available (along with RHEL engineer packaged iotop related RPMs) are available athttp://people.redhat.com/jolsa/iotop/ ; while the patches there aren’t large, it isn’t as easy as just turning on the kernel config option.
Greg Smith
3 Jan 11 at 3:02 pm
For those trying to get Guillaume Chazarain’s sophisticated iotop to work, try my humble little topio program as a workaround:
http://yong321.freeshell.org/freeware/pio.html#linux
Over the past 7 years, I ported my program to multiple OS’es. The Linux version was done only a few months ago.
Yong Huang
3 Jan 11 at 3:48 pm
iodump appears to have a bug in it, as it always reports a 0 for “DIRTY”. Unfortunately, I am not a perl or regex expert and after spending several hours with this script cannot seem to get a valid value for DIRTY count. The reason I believe DIRTY count is important is the system I’m testing has an ext3 file system with kjournald doing reads and writes on behalf of processes. So if a process “dirty” requires a kjournald write, that won’t show up without the DIRTY value. Plus, the key measure I’m looking at right now is per process startup resource usage, and a command line linux tool such as iostat that can’t be run until well after startup has completed won’t help. Was hoping to use iodump to process the kernel log to crunch the startup data. Thanks!
Chris Rizzo
10 Feb 12 at 5:12 pm
Ahhh… I believe I’ve found the bugs in the script. The dirtied regex should be:
m/(S+)((d+)): (dirtied) inode (d+) (.*?) on (S+)/
And line 69 should be $task->{‘dirtied’}, not $task->{‘dirty’} (given the following input):
Feb 9 20:20:23 XRX_0000AA9E8ABA kernel: [ 14.974373] klogd(242): dirtied inode 47718 (klogd.pid) on sda5
Chris Rizzo