Scratching that vmstat plotting itch

I’ve been working on an article around vmstat-like tools and I started looking for ways to plot the data. There are some reasonable articles on using something like awk and gnuplot to create plots but I wanted something a little more self-contained that would produce a quick report with all of the data plotted. I did see one tool that was on-line only (you had to upload your data) but that was too much of a pain to use, so in true Clusterbuffer fashion I wrote my own.

It’s not a sophisticated tool by any stretch – it’s just some Python code to parse the data and plot it using matplotlib. Go to this page to read more about it and scroll down to the bottom to download the code. You can also see a simple example here.

Please read through the write-up since there are some limitations to the code that require vmstat to be run with certain flags. There are also vmstat options that it can’t plot because the output is not time dependent.


NFSiostat_plotter V4

I made an update to nfsiostat_plotter_v3 so that the iostat information, which is used for gathering the CPU usage, is optional. This means you don’t have to run iostat while you run nfsiostat. But it also means you won’t get to see the CPU usage plots.

Please go to this page for more details

IOSTAT Plotter and NFSiostat plotter updates (V3!)

I’ve been updating both nfsiostat_plotter and nfsiostat_plotter. Both versions are now “v3”. There are a few new features in both:

  • Normally nfsiostat doesn’t capture the CPU utilization but this can be an important part of analyzing NFS client performance. In this version of the code you need to also run iostat while you run nfsiostat so that the CPU utilization is captured. With this version of code, you have to run iostat and run iostat_plotter_v3.py to produce the iostat output (which is read by nfsiostat_plotter).
  • For both iostat_plotter and nfsiostat_plotter, the legend has been moved outside the plot to the top right
  • For both iosta_plotter and nfsiostat_plotter, the size of the legend has been shrunk so you can plot more NFS file systems (up to about four before the legend lables start to overlap). Iostat_plotter can handle about 8 devices.
  • For nfsiostat_plotter, the command option “-c” was added so you could combine the NFS file systems onto a single plot (total of 4 plots)
  • For iostat_plotter, the command option “-c” was added so you could combine the devices onto a single plot (total of 4 plots)
  • For iostat_plotter, the code can read either sysstat v9.x or sysstat v10.x format for iostat. This is very important since CentOS and RHEL ship with a very old sysstat v9.x package.
  • For both iostat_plotter and nfsiostat_plotter, the legend labels are auto-sized based on the string length (it’s a heuristic algorithm).
  • For both iostat_plotter and nfsiostat_plotter, a simple “-h” option as added that produces a small help output.

Go to this page for iostat_plotter_v3 and this for nfsiostat_plotter_v3. You will see links to the new code. But just in case, here are the links.

IOSTAT Plotter V2 – Update

I’ve been updating nfsiostat_plotter and in the process I found a few bugs in iostat_plotter V2.

The first bug was in the way it kept time. If the time stamp ended in “PM” the code adds 12 hours so everything is based on a 24 hour clock. The problem is that if the iostat output is run just after noon, the time reads something like “12:49 PM” which then becomes 24:29. This bug has been fixed.

The second bug was in the plotting routines when doing “combined” plots where all devices are plotted on the same graph. It’s hard to explain but when there two or three plots, one above the other, sometimes the grid didn’t get plotted. And/or one of the graphs was longer than the others. This has been corrected (I hope).

The new version can be found here. The code has the same name as before (sorry about that). Just download the code and run it like you did before.

IOSTAT Plotter V2

I have had several requests for changes/modifications/updates to iostat_plotter so I recently found a little time to update the code. Rather than just replace I created, iostat_plotter_v2.

The two biggest changes made in the code are: (1) moving the legends outside the plots to the top right hand corner, and (2) allowing all of the devices to be included on the same set of plots. The first change makes look nicer (I think). I tried to maximize the width of the plot without getting too crazy. I also shrunk the text size on the legend so you could get more devices in the plots. I think you can get about 12 devices without the legend bleeding over to the plot below it.

The second change allows you to put all of the devices on the same plot if you use the “-c” option with the script. In the previous version you got a set of 11 plots for each device which allows you to clearly examine each device. If you had five devices you got a total of 55 plots (5 devices * 11 plots per device). Using the “-c” option you will now get 11 plots with all of the devices on each of the appropriate plots.

I hope these couple of features are useful. Please le me know if they are or if you want other changes. Thanks!

Linux 3.1: Steady Storage

The gestation period for the 3.1 kernel was fairly long for various reasons, but it’s out and there are some steady storage updates but nothing earth shattering. But don’t mistake this for complacency, the storage bits of the kernel are still progressing very nicely.

3.1 – Me Develop You Long Time

The development time of the 3.1 kernel was a bit longer than usual starting with the 3.0 release on from July 21 to the 3.1 release on Oct. 24 of this year. Three months is a little longer than usual but Linux was on vacation during the development and kernel.org had a few security issues. But the intrepid kernel developers persevered and we have a shiny new Linux kernel. Let’s take a look at the storage aspects of the 3.1 kernel starting with the fan favorite, file systems.

File Systems

There were some nice new features added to the 3.1 kernel around file systems.

The ext3 file system, while some outdated, is still in very heavy use. In the 3.1 kernel, a very nice feature was added to the default options for ext3. The option that is now default in et3 is write barriers. You can read more about write barriers here but in general write barriers ensure that the data is written to the disk at key points in the data cycle. In particular, write barriers happen before the journal commit to make sure all of the transaction logs are on the disk. Write barriers will impact performance for many workloads, but the trade-off is that you get a much more reliable file system with less chance of losing data. Ext3 had write barriers for some time but they were never turned on by default even though most distributions turned them on. So the kernel developers decided in the 3.1 kernel that they would turn on write barriers by default. While I am a performance junky, I’m an even bigger junky for making sure I don’t lose data.

I’m truly surprised by how many people who know Linux and file systems well enough still do silly things such as making the command, “ls -l” part of a script or put millions of 4KB files in a single directory. Performing a simple “ls” command, or the more feared, “ls -l”, puts a tremendous strain on the file system forcing it to walk a big chunk of the file system and look up inodes to gather metadata. A big part of this process is the use of the readdir() function.

In the 3.1 kernel, Josef Bacik from Red Hat, created a patch that improves readdir() performance, particularly for “ls” commands that follow readdir() with an stat() command. The patch improved performance on a simple, but long running script that uses “ls” by 1,300 seconds (22 minutes). Here is a plot of the pattern of the script before the patch, and here is a plot of the same script after the patch. The differences between the two are remarkable.

The second patch to btrfs, while somewhat complicated because it deals with the idea of locking during metadata operations, results in significantly better read performance and in most cases improved write performance. However, there are cases, such as those for dbench, where the write performance suffers a bit. But overall, it is a good change for btrfs.

Everyone loves NFS (if you don’t, you should), because it’s the only file system standard. NFS is NFS is NFS. It is the same protocol for Linux as for OSX as for HPUX, as for Windows, etc. So it allows one to share data between different systems even if they have different operating systems. Plus it is a well known file system that is fairly easy to configure and operate and has well-known error paths.

In the 3.1 kernel, there was a patch to NFS, actually to the v4.1 standard of NFS (sometimes called pNFS), that added IPV6 support. This patch is fairly significant because as pNFS becomes a reality (yeah – it’s been a long road), some people are going to want to use IPV6, particularly in government areas. So this patch is a dandy one for NFS.

The XFS developers have been keeping a steady pace of great developments. In the 3.1 kernel they added some performance enhancements. The details are very involved, but if you want to read the kernel commits, you can see the first one here and the second one here.

The venerable reiserfs file system is still used in many places. In the 3.1 kernel, a simple patch was included to make the default mount option for reiserfs to barrier=flush. This will have an impact on existing reiserfs file systems that are upgraded to the 3.1 kernel.

HFSPlus (HFS+) is a file system developed by Apple. It is used on a variety of Apple platforms including iPods. In the 3.1 kernel, a 2TB limitation on HFS+ was changed to a dynamic limit based on the block size. While HFS+ may not be used as a primary file system for most Linux users, having the ability to mount and interact with Apple file systems can be very useful.

One of my favorite file systems in the Linux kernel is SquashFS (I think I’ve said this before). In past kernels, squashfs supported the XZ and LZO compression methods in the kernel, in addition to the default ZLIB compression method, giving it access to three different compression methods. It was decided to drop the ZLIB support in the 3.1 kernel since there are two other compression options. Plus dropping support for ZLIB reduces the kernels size which is important for people using Linux for embedded applications.

Block Layer Patches

File systems are cool and people love to talk about them but there are other aspects to Linux storage that are just as key. One important aspect is the block layer. In the 3.1 kernel there were several important patches to the block layer that you should be aware of.

One patch that may seem esoteric but is more important than you think, added the ability to force the completion of an IO operation to go to the core (cpu) that request the operation. The kernel has the ability to move processes around the cores in a system. For IO operations, the block layer usually has the concept of completing an IO operation to a core that is on the same socket as the requesting process (using the blk_cpu_to_group() function). Sometimes you want the requesting process to also get the completion notification. This patch allows you to set capability on a per block device basis such as,

echo 2 > /sys/block//queue/rq_affinity

There were a couple of performance tuning patches for the CFQ IO scheduler that were pretty detailed and beyond the scope of this quick article. But you have to love performance additions.

The third set of changes to the block layer focused on the device mapper (dm) portion of the block layer. The first patch that is significant is the addition of the capability of supporting the MD (multi-device) RAID-1 personality using dm-raid. This is not a insignificant patch since the MD RAID-1 has some unique features. With this patch you can now use the same features but using dm-raid.

The second dm patch adds the ability to parse and use metadata devices with dm-raid. This is significant because the metadata information allows dm-raid to correctly reassemble disks in the correct order when booting or if a disk fails or there is some other fault. This is an extremely important task that otherwise you have to do by hand (yuck).

VFS Layer

If you have looked at the kernel patches for the last several kernels you will realize that there has been a large effort around improving the scalability of the VFS layer. The 3.1 kernel added some additional patches to improve scalability (and performance). The details are there if you want to read them but they are pretty deep.


iSCSI is becoming more popular as a replacement for Fibre Channel (FC) networks since you just use Ethernet networks. In the 3.1 kernel the current iSCSI implementation, called STGT, was declared obsolete after the inclusion of the linux-iscsi.org (LIO) SCSI target was included in the kernel. LIO is a full featured in-kernel implementation o the iSCSI target mode (RFC-3720) but was not the only in-kernel iSCSI target considered. A second set of patches, called SCST was also considered and I guess the discussion around LIO vs. SCST was pretty rough. For a more in-depth discussion you can read the lwn article here.


The 3.1 kernel was fairly quiet from a storage perspective but the thing I like most about it is that steady progress was made on the kernel. If you recall there was a period of time when storage development in the kernel was stagnate. Then in the later 2.6.x series it start picking up with a huge flurry of development, new file systems, scalability improvements, and performance improvements. So the fact that storage is still being developed in the kernel illustrates that it is important and work is still on-going.

There were some file system developments of note, particularly if you are upgrading an existing system to use the 3.1 kernel. There were also some developments in the block layer (if block layer patches go into the kernel you know they are good quality because no one wants to cause problems in the block layer) and the VFS layer had some further scalability improvements. And finally for people using iSCSI, there was a new iSCSI target implementation included in the kernel, so if you use iSCSI targets on Linux, be sure you test these out thoroughly before deploying things in production.

RIP Nedit

I have been using Nedit for many years for just about every editing need. However, a recent upgrade to Kubuntu 11.04 illustrated that Nedit was on it’s last leg and it was time to switch. RIP Nedit.


I having been using Nedit for many years for just about every editing need I have. I use for writing code (it’s really great in this aspect), and I write these articles using Nedit (I have been for a very long time). it’s really easy to use, I love the split screen capability (but you need lots of vertical space for it to be really effective), I like the ability to edit multiple files and switch between them with a tab (yes – this is an extremely common capability but many years ago this didn’t exist), and I liked the auto-indentation feature (great when writing Python code). Overall I just found it to be a great editor so I started using it full time many years ago on IRIX and continued using it on Linux.

I started to use more features in Nedit and it just became comfortable for me to use. Most people will tell you that once they find a tool that scratches their proverbial itch they are very reluctant to change. I fall into this category. My wife even makes fun of me saying that the reason I don’t switch to anything else is because I don’t like change. Sorry sweety – in this case that isn’t the correct observation. The reason I don’t switch is that it would take me a long time to learn a new tool and become as proficient on it as with Nedit. Is it worth it to go through this process expending time and effort to learn a new editor just because it’s new? It’s highly unlikely that this new tool will make me more productive than with Nedit, so why switch? Switching for the sake of switching is good in some circumstances, but I don’t think this is one of them.

So I soldiered on for many years using Nedit and it worked just great. Whomever built the packages or binaries for the various distro I used (mostly CentOS, Scientific Linux, and Kubuntu) did a great job, presumably using the Lesstif library to build Nedit since it’s built on the Motif toolkit. I didn’t look at how they built the binaries, nor did I really want to know – I was just happy to have Nedit around. And, by the way, if you reading this and you built these binaries and packages, thanks very much. Count me as a happy and very appreciative customer.

Then I upgrade my laptop to Kubuntu 11.04 and installed Nedit. Then I noticed a small problem – when I tried copying and pasting a section of text or code, it would not paste. Instead I got the following message in the terminal window:

NEdit warning:
XmClipboardInquireLength() failed: clipboard locked.

A little googling found recent post that indicates that Nedit isn’t really working on Ubuntu anymore. I’m not sure if it’s a Nedit problem or a Motif library (presumably Lesstif or OpenMotif) but the result is that Nedit no longer works for me on Kubuntu 11.04 (Note: it works fine on CentOS 5.5 and older Ubuntu versions such as 8.04, but not the more recent versions). The only solution is to save any work, exit from Nedit, and start again.

If you couple this problem with comments from my friend Joe Landman, and I had to start looking for a new editor.


I tried all kinds of editors including Kate, Kedit, geany, JuffEd, and gedit. I’m a KDE kind of guy because I really loath Gnome (and the recent interfaces haven’t made me want to use it) but then again, the whole KDE 4.0 thing was a mess and not until KDE 4.6 or so, was it useable again.

Kate was fine and I used for some work but I found the interface to be a bit clunky. The same for Kedit, although Kedit moved up my list fairly rapidly. I ended up settling on gedit.

Gedit doesn’t have all the features I want especially being able to split the window so I can see two different parts of the code at the same time, but given the choice between an editor that doesn’t really work but has all the features, and an editor that works but doesn’t have all the features, I think I’ll chose the later. This is what gedit does for me.

If you have any ideas or suggestions for visual editors please don’t hesitate to suggest something.