An Introduction to Gnuplot

Introduction

Gnuplot is an incredibly powerful and useful utility for presenting data in graphical format. In this post I’ll cover a very simple usecase that involves graphing the size of a database over a period of time.

Installing

On Linux use your normal package manager, e.g.

# Debian based...
apt-get install gnuplot

# Redhat based...
yum install gnuplot

The easiest way to install gnuplot on OS X is using brew. Gnuplot requires X11 to display graphs so to install it use,

brew install gnuplot --with-x11

If X11 support isn’t available gnuplot will fail to display graphs with the error,

WARNING: Plotting with an 'unknown' terminal.
No output will be generated. Please select a terminal with 'set terminal'.

If this happens uninstall and then reinstall with X11 support.

Basic Usage

In the example below I’ll use a data file called ‘db_size.dat‘. Each line in this file records the size of a database at an hourly interval.

The most basic gnuplot command to graph this data would be,

gnuplot -p -e "plot 'db_size.dat'"

The -p parameter (short for persist) leaves the window containing the graph open after gnuplot finishes.

The -e parameter is the command to pass to gnuplot. The command “plot ‘db_size.dat'” means plot a graph using the data in db_size.dat.

Running this command will open a a window with this graph.

Graph 1

Depending on your requirements this simple graph may be absolutely fine. However if you’d like to polish it up there are a few changes we can make.

First of all lets give the graph a title.

gnuplot -p -e "set title 'Database Size'; plot 'db_size.dat'"

Graph 2
At this stage the command line is getting a little long. This is where gnuplot command files come into play. Each file is basically a list of commands.

This command file is called ‘db_size.gnuplot’,

set title 'Database Size'
plot 'db_size.dat'

The command file name is passed to gnuplot as a parameter,

gnuplot -p db_size.gnuplot

Lets add axis labels and give the key a name.

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (bytes)"
plot 'db_size.dat' title 'Database Size'

Graph 3

The dots can be difficult to follow when the they’re spaced out so lets use a line instead.

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (bytes)"
plot 'db_size.dat' title 'Database Size' with line

Graph 4

The last change I’d like to make is to make the Y-axis scale more readable by converting bytes to gigabytes.

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (GB)"
plot 'db_size.dat' using ($1/1000000000) title 'Database Size' with line

Graph 5

Finally lets save the graph to a PNG rather than displaying it on screen.

set term png
set output "db_size.png"
set title 'Database Size'
set xlabel "Time"
set ylabel "Size (GB)"
plot 'db_size.dat' using ($1/1000000000) title 'Database Size' with line

This saves the graph to a file called db_size.png in the current directory.

Going a Little Further

In the previous example there was one data value recorded at fixed intervals, i.e. hourly. But what if the same data is collected at irregular intervals? To reliably graph this data we need to record the time each data point is recorded, db_size_with_time.dat. In this file the first column is seconds since the epoch. Gnuplot can parse this using the ‘set xdata’ command. Here’s the full command file,

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (GB)"
set key off
set xdata time
set timefmt "%s"
set xtics format "%H"
plot 'db_size_with_time.dat' using 1:($2/1000000000) with line

‘set key off’ turns off the key. Its not very useful in this graph.

‘set xdata time’ parses the X axis data as a time.

‘set timefmt “%s”‘ means read time values as seconds since the epoch.

‘set xtics form “%H”‘ means write the hour to the X axis

Database Size with Time

This post barely scratches the surface of what can be achieved using gnuplot. It’s really an amazing piece of software.

For more information:

The Irish Constitution Git Repository

The Irish Constitution and each of it’s amendments are published by the Office of the Attorney General on the Irish Statute Book website. To date there have been thirty-three amendments proposed with twenty-seven being passed.

Each amendment describes the change being proposed like this (from the second amendment in respect of a change being proposed to article 25),

In sub-section 1° of section 2, the deletion of the word “five” and the insertion in its place of the words “the fifth”, and the deletion of the words “seven days” and the insertion in their place of the words “the seventh day

While its accurate it’s not very easy to see the impact of the change in the context of the document.

In the software engineering world we store code, which is just text, in source control. Using the tools provided by the source control system we can compare two versions of a document to see how they differ.

To get to the point where I could compare different versions of the constitution I assembled a text version of the document and created a commit for each amendment. The end result is this git repository, https://github.com/adrian/irish-constitution.

Every amendment up to and including amendment thirty-three is included.

For the sake of completeness I included the transitory articles 51 to 64 and 34a. Rather than including the transitory sub-sections 29.7.3, 29.7.4 and 29.7.5 (from amendment 19) I made the changes directly to articles 2 and 3.

Each commit of constitution.md represents an amendment. By opening any of these commits we can see how that amendment changed the constitution. For example, amendment thirty-three looks like this.

The left column is the constitution before amendment thirty-three, the right is after amendment thirty-three. Red lines are deleted or changed with dark red words deleted. Green lines are added or changed with dark green words added. By showing the two versions of the document, before and after, and highlighting the changes the effect of the amendment should be easier to understand.

To round it off there are two pending changes (pull requests) for each of the upcoming referendums on the 22nd May 2015,

 

Installing Debian 7.1 “Wheezy”

Every time I install Linux, no matter what the distribution, I have the same problems; configuring video and audio. When I installed Debian 7.1 recently I documented the steps I took. Hopefully you’ll find something of use here.

My Hardware

Video: Advanced Micro Devices [AMD] nee ATI RV710 [Radeon HD 4350]
Sound: Advanced Micro Devices [AMD] nee ATI RV710/730 HDMI Audio [Radeon HD 4000 series]
Monitor: Dell SP2208WFP (with built in webcam)
WebCam: OmniVision Technologies, Inc. Monitor Webcam

Video and sound devices can be determined using lspci. The Webcam is a USB device so it can be determined using lsusb.

Video

For regular use the opensource Radeon display driver is considered adequate. If you’re a gamer or just want to squeeze every bit of performance out of your card then you might consider the AMD proprietary display driver. For my needs the opensource driver is fine.

On the first boot GNOME 3 failed to load. The installer recognized my video card and installed the correct package, xserver-xorg-video-radeon, but as documented on the wiki my card requires proprietary firmware. This firmware is available in non-free. The steps to install it are,

  1. Open up Synaptic Package Manager and select Settings, Repositories from the menu
  2. Tick the checkbox next to Non-DFSG-compatible Software (non-free)
  3. Click Close and then the Reload toolbar button
  4. Search for and install firmware-linux-nonfree
  5. Reboot

GNOME 3 should start successfully after the reboot.

Audio

When I didn’t hear any sound I thought I was going to be in for a world of pain with sound drivers, ALSA and PulseAudio. Luckily all I needed to do was select the right output device. My system has two sound devices, an onboard Intel device and the sound device on the AMD video card. My monitor is connected by HDMI. I have a soundbar on my monitor drawing it’s signal from the HDMI cable. All I needed to do was,

  1. Open up the System Settings
  2. Click the Sound icon
  3. In the Output tab select the HDMI audio device, i.e. RV710/730 HDMI Audio [Radeon HD 4000 series] Digital Stereo (HDMI)

Update (Aug-2014): HDMI audio is disabled by default in the kernel/audio driver. To enable it,

  1. Edit /etc/default/grub and append “radeon.audio=1” to the GRUB_CMDLINE_LINUX_DEFAULT variable.
  2. Open a terminal execute sudo update-grub
  3. Reboot

Source: No sound on HDMI with Radeon driver

Webcam

My monitor has a built in Webcam. It’s an OmniVision Technologies, Inc. Monitor Webcam. I managed to get it working but it’s a bit hit and miss. When then uvcvideo kernel module is loaded it should create the device /dev/video0. Sometimes it does, sometimes it doesn’t. To get it to work is a matter of removing the module and reloading. After two or three attempts it usually works. Not very satisfying but I don’t use it very often so I’m not that bothered.

$ sudo modprobe -r uvcvideo     # unload the module
$ sudo modprobe uvcvideo        # load the module

Once the module loads successfully and creates the /dev/video0 device you can use the Cheese application to test it.

Microphone

The mic didn’t appear to be working initially but as with audio out all I needed to do was select the right device and turn up the volume. I use the mic on my monitor (attached to the webcam).

  1. Open up the System Settings
  2. Click the Sound icon
  3. Select the Input tab
  4. Select the Monitor Webcam Analog Stero device
  5. Turn the volume up to 100%

After getting the hardware working these are the applications I installed.

Flash Player

The Flash Player package is avaiable from contrib.

  1. Open up Synaptic Package Manager and select Settings, Repositories from the menu
  2. Tick the checkbox next to DFSG-compatible Software with Non-Free Dependencies (contrib)
  3. Click Close and then the Reload toolbar button
  4. Search for and install flashplugin-nonfree
  5. Restart your browser if you

Reference: FlashPlayer on the Debian Wiki

Java Browser Plugin

Simply install the package icedtea-6-plugin.

gThumb

gThumb is a photo manager. I prefer it over the default, Shotwell, because it feels quicker and more lightweight. I already organise my photos in a year, month, day folder structure so I have no need for anything fancy.

While gThumb itself is very easy to install (the gthumb package) it was failing to properly list all the photos and videos in a directory containing one or more videos. The problem was it couldn’t find the image it uses to surround videos (to give the reel effect). The fix is simple,

$ sudo ln -s /usr/share/gthumb/ui/filmholes.png /usr/share/filmholes.png

Reference: Redhat Bug 842183 – Gthumb breaks file list on movies

Android Tools

The Android Tools are used when developing Android applications. After installing the Android SDK I found adb wouldn’t work. When I ran it it gave a “file not found”. The problem is down to missing shared libraries. Here are the steps I took to fix it,

$ sudo dpkg --add-architecture i386
$ sudo aptitude update
$ sudo aptitude install libstdc++6:i386 libgcc1:i386 zlib1g:i386 libncurses5:i386

Reference: Installing Android SDK tools on Debian Wheezy

IFTTT – An Alternative to Feed Aggregators

Since Google Reader was retired I’ve been on the lookout for something a bit different. I tried out a few services but none of them really fit in with my way of doing things. When I thought about it all I really wanted was a service that would send me an email when a new entry was posted to a blog I liked. Luckily that’s exactly the kind of thing IFTTT does.

IFTTT is a web service that allows you to create little “programs”. These programs, or recipes as they’re called, carry out an action following some trigger event. Each recipe has the same structure; “if this happens then do that”. The “this” piece is called the trigger. The “that” piece is called the action.

Example triggers,

  • I tweet a message
  • I update my Facebook status
  • I upload a photo to Instagram
  • A new entry is posted to a blog

Example actions,

  • Send me an SMS
  • Upload a photo to Facebook
  • Add an event to my Google Calendar
  • Send me an email

See About IFTTT for a more comprehensive description.

Actions and triggers are organized into channels, e.g. Facebook, Dropbox, Email, Feed (i.e. blog feed), etc.

Using the “New feed item” trigger in the Feed channel and the “Send me an email” action in the Email channel I put together a recipe for each blog I wanted to subscribe to. Here’s what my XKCD recipe looks like,

XKCD Recipe

The email subject has the blog name and entry title. For the body I chose to include a link to the entry rather than the full text. Here’s what an email from IFTTT looks like,

XKD Monster

I’ve subscribed to number of blogs this way. What I really like is getting email notifications pushed to me rather than having to use a blog aggregation tool. I use email as a sort of TODO list. This way I can treat new blog posts as something I need to do. Once I have the email I can come back to it whenever I want.

As good as IFTTT is there are some rough edges for my particular use case. I wouldn’t say they’re problems though because IFTTT is meant to be a general service.

  1. If I want to subscribe to a new blog it’s a bit of a nuisance having to create a whole new recipe from scratch. It would be great if I could create a new recipe based on another one. That way all I’d need to change is the recipe’s name, the blog feed URL and the mail subject (which includes the blog title). The positive side of this is that it stops me subscribing to blogs willy-nilly. I prefer a few quality subscriptions.
  2. I’d like to be able to export my recipes. Why? I think it’s the programmer in me. Now I have all these recipes I feel like they should be under version control.
  3. An IFTTT API would be great. If there was an API to create a recipe I could create a browser extension to automatically create a “Subscribe to RSS” recipe while on a blog.

So that’s my experience. It works for me.