NFSiostat_plotter

Introduction

This tool, nfsiostat_plotter.py simply takes the output from nfsiotstat, parsers the data to some degree, and creates plots and an HTML report. The intent is to allow plot the time history of the NFS IO rather than have to stare at numbers flashing by on the command line.

There are several versions of nfsiostat_plotter. The first version, nfsiostat_plotter_article.py below corresponds to the version used in the article in HPC-Admin magazine, and can be downloaded. The second version, nfsiostat_plotter.py is an enhanced version with the ability to import data from the enhanced version of iostat_plotter.py (link). It also has a serialization capability (“pickle” in Python-speak) that allows you to save the data structures for use elsewhere.

Both versions are covered under the GPLv2 license.


Article Version:

This version is labeled “nfsiostat_plotter_article.py”. This corresponds to the version in the HPC-Admin Magazine article.


You need to get the nfsiostat output using the command

[laytonj@home8 NFSIOSTAT]$ nfsiostat -h -m -t 1 100 > nfsiostat.out

where “-h” is for a more human readable output, “-m” is for the data to be in MB, “t” is to include time in the output, and “1 100” tells nfsiostat to use “1” second intervals and “100” means to gather data for 100 samples (or 100 sceonds in this case).

After nfsiostat is run, then you can run nfsiostat_plotter_article using the command,

[laytonjb NFSIOSTAT]$ nfsiostat_plotter_article.py nfsiostat.out

where “nfsiostat.out” is the output from nfsiostat. The code is written in Python (obviously) and uses the shlex, time, os, and matplotlib modules.

When nfsiostat_plotter_article is done it will create a subdirectory “HTML_REPORT” that contains the plots and an html file “report.html”. Open that html file in a browser or word processor and you will see the plots and a small write-up about them. Feel free to modify the code but please send back changes.

An example of the html output is here. You can download the file using this link (sorry – I have to use dropbox because WordPress does not allow .py files).


Enhanced Version


This version can be found on page.

Advertisements

11 Responses to NFSiostat_plotter

  1. José Gonzalez says:

    Hi there,

    I have downloaded this app but it doesn’t seem to work with my nfsiostat output.

    In fact, the command you suggest running to collect the data, “nfsiostat -h -m -t 1 100 > nfsiostat.out” seems to be incorrect, as nfsiostat does not have these flags.
    It seems to be (as stated on the header of the script) that it requires the output of iostat on a per device basis (-n to show nfs statistics) “$ iostat -c -d -x -t -m /dev/sda 1 100 > iostat.out”

    Unless of course we have different versions of nfsiostat. Though I didn’t know there was more than one, with such different flags… Here’s mine:
    Options:

    version 0.2

    –version show program’s version number and exit
    -h, –help show this help message and exit

    Statistics Options:
    File I/O is displayed unless one of the following is specified:

    -a, –attr displays statistics related to the attribute cache
    -d, –dir displays statistics related to directory operations
    -p, –page displays statistics related to the page cache

    Display Options:
    Options affecting display format:

    -s, –sort Sort NFS mount points by ops/second
    -l LIST, –list=LIST
    only print stats for first LIST mount points

    Any help would be much appreciated.

    Thank you.
    José

    • Jose,

      I’m not sure what is happening but I suspect the problem lies with the version of nfsiostat you are using. I used verions 10.x of sysstat (http://sebastien.godard.pagesperso-orange.fr/). I’m thining you are using version 9.x. If you type “nfsiostat -V” what does the output say? If it says “sysstat version 9…” then you need to upgrade your version of sysstat.

      It is really easy to build if you have to. I used the command “./configure –pref=/usr” and then I did “make” following by “make install” (as root) and it just overwrote the 9.x version.

      I hope this helps!

      Jeff

  2. Dow Hurst says:

    The download link leads to iostat plotter script not the nfsiostat link. would you update?

  3. Pingback: IOSTAT Plotter V2 – Update | clusterbuffer

  4. Pingback: IOSTAT Plotter and NFSiostat plotter updates (V3!) | clusterbuffer

  5. Dan Lindh says:

    Hi!

    Found your code really interesting but I get a strange error about pickle.

    I run
    # iostat -n -h -m -t 1 100 > nfsiostat.out
    followed by
    ./nfsiostatplotter.py nfsiostat.out but it just gives me the following.

    input filename: nfsiostat.out
    I like to unpickle it, unpickle it
    Traceback (most recent call last):
    File “./nfsiostatplotter.py”, line 709, in
    pickle_file = open(‘iostat_file.pickle’, ‘r’);
    IOError: [Errno 2] No such file or directory: ‘iostat_file.pickle’

    I guess its a simple thing somewhere 😦

    • Dan,

      Currently you have to run iostat along with nfsiostat. The reason is that nfsiostat doesn’t gather CPU stats yet nfsiostat does and some of the plots in nfsiostat use CPU usage. Then you run iostat first which produces the pickle file.

      I thought I mentioned this but I may missed it or it may be understated. I will come up with a new version of nfsiostat that will write out an error if it can’t find the pickle file. A future vresion will adjust the plots based on whether the pickle exists or not.

      My apologies for the confusion.

      Jeff

      • Dan,

        I just uploaded nfsiostat_plotter_v4. It allows you to create plots without having to run iostat along with nfsiostat. Of course you won’t get the CPU usage information but it does allow you to at least plot the nfsiostat information. Enjoy!

        Jeff

  6. David Sloboda says:

    Could you please provide the link to nfsiostat_plotter_v4 ?
    Could you please provide the URL to your upload directory?
    Thank you,

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: