An Introduction to Gnuplot

Introduction

Gnuplot is an incredibly powerful and useful utility for presenting data in graphical format. In this post I’ll cover a very simple usecase that involves graphing the size of a database over a period of time.

Installing

On Linux use your normal package manager, e.g.

# Debian based...
apt-get install gnuplot

# Redhat based...
yum install gnuplot

The easiest way to install gnuplot on OS X is using brew. Gnuplot requires X11 to display graphs so to install it use,

brew install gnuplot --with-x11

If X11 support isn’t available gnuplot will fail to display graphs with the error,

WARNING: Plotting with an 'unknown' terminal.
No output will be generated. Please select a terminal with 'set terminal'.

If this happens uninstall and then reinstall with X11 support.

Basic Usage

In the example below I’ll use a data file called ‘db_size.dat‘. Each line in this file records the size of a database at an hourly interval.

The most basic gnuplot command to graph this data would be,

gnuplot -p -e "plot 'db_size.dat'"

The -p parameter (short for persist) leaves the window containing the graph open after gnuplot finishes.

The -e parameter is the command to pass to gnuplot. The command “plot ‘db_size.dat'” means plot a graph using the data in db_size.dat.

Running this command will open a a window with this graph.

Graph 1

Depending on your requirements this simple graph may be absolutely fine. However if you’d like to polish it up there are a few changes we can make.

First of all lets give the graph a title.

gnuplot -p -e "set title 'Database Size'; plot 'db_size.dat'"

Graph 2
At this stage the command line is getting a little long. This is where gnuplot command files come into play. Each file is basically a list of commands.

This command file is called ‘db_size.gnuplot’,

set title 'Database Size'
plot 'db_size.dat'

The command file name is passed to gnuplot as a parameter,

gnuplot -p db_size.gnuplot

Lets add axis labels and give the key a name.

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (bytes)"
plot 'db_size.dat' title 'Database Size'

Graph 3

The dots can be difficult to follow when the they’re spaced out so lets use a line instead.

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (bytes)"
plot 'db_size.dat' title 'Database Size' with line

Graph 4

The last change I’d like to make is to make the Y-axis scale more readable by converting bytes to gigabytes.

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (GB)"
plot 'db_size.dat' using ($1/1000000000) title 'Database Size' with line

Graph 5

Finally lets save the graph to a PNG rather than displaying it on screen.

set term png
set output "db_size.png"
set title 'Database Size'
set xlabel "Time"
set ylabel "Size (GB)"
plot 'db_size.dat' using ($1/1000000000) title 'Database Size' with line

This saves the graph to a file called db_size.png in the current directory.

Going a Little Further

In the previous example there was one data value recorded at fixed intervals, i.e. hourly. But what if the same data is collected at irregular intervals? To reliably graph this data we need to record the time each data point is recorded, db_size_with_time.dat. In this file the first column is seconds since the epoch. Gnuplot can parse this using the ‘set xdata’ command. Here’s the full command file,

set title 'Database Size'
set xlabel "Time"
set ylabel "Size (GB)"
set key off
set xdata time
set timefmt "%s"
set xtics format "%H"
plot 'db_size_with_time.dat' using 1:($2/1000000000) with line

‘set key off’ turns off the key. Its not very useful in this graph.

‘set xdata time’ parses the X axis data as a time.

‘set timefmt “%s”‘ means read time values as seconds since the epoch.

‘set xtics form “%H”‘ means write the hour to the X axis

Database Size with Time

This post barely scratches the surface of what can be achieved using gnuplot. It’s really an amazing piece of software.

For more information:

The Irish Constitution Git Repository

The Irish Constitution and each of it’s amendments are published by the Office of the Attorney General on the Irish Statute Book website. To date there have been thirty-three amendments proposed with twenty-seven being passed.

Each amendment describes the change being proposed like this (from the second amendment in respect of a change being proposed to article 25),

In sub-section 1° of section 2, the deletion of the word “five” and the insertion in its place of the words “the fifth”, and the deletion of the words “seven days” and the insertion in their place of the words “the seventh day

While its accurate it’s not very easy to see the impact of the change in the context of the document.

In the software engineering world we store code, which is just text, in source control. Using the tools provided by the source control system we can compare two versions of a document to see how they differ.

To get to the point where I could compare different versions of the constitution I assembled a text version of the document and created a commit for each amendment. The end result is this git repository, https://github.com/adrian/irish-constitution.

Every amendment up to and including amendment thirty-three is included.

For the sake of completeness I included the transitory articles 51 to 64 and 34a. Rather than including the transitory sub-sections 29.7.3, 29.7.4 and 29.7.5 (from amendment 19) I made the changes directly to articles 2 and 3.

Each commit of constitution.md represents an amendment. By opening any of these commits we can see how that amendment changed the constitution. For example, amendment thirty-three looks like this.

The left column is the constitution before amendment thirty-three, the right is after amendment thirty-three. Red lines are deleted or changed with dark red words deleted. Green lines are added or changed with dark green words added. By showing the two versions of the document, before and after, and highlighting the changes the effect of the amendment should be easier to understand.

To round it off there are two pending changes (pull requests) for each of the upcoming referendums on the 22nd May 2015,

 

Installing Android Studio on Debian Wheezy

Android Studio is the new IntelliJ IDEA based Android IDE. It succeeds the current Eclipse based IDE.

Android Studio requires Oracle Java 1.6 or later. It won’t work with OpenJDK.

Installing Oracle Java 1.7

These instructions describe how to install Oracle Java using java-common. java-common only works with Java 1.6 or 1.7. It doesn’t work with Java 8.

  • Download the latest 1.7 JDK from http://www.oracle.com/technetwork/java/javase/downloads/index.html. To find the tarball follow the links “Previous Releases – Java Archive” -> “Java SE 7” -> “Java SE Development Kit 7u..”. You’ll need the tarball not the RPM, e.g. jdk-7u67-linux-x64.tar.gz. Note: you’ll need an Oracle account to download archive versions of Java (those < Java 8).
  • Install java-common

    # apt-get install java-common

  • Create a deb file using the Java tarball,
    $ make-jpkg <downloaded tarball>
  • Install the resulting package (your deb file name will likely be different),
    # deb -i oracle-j2sdk1.7_1.7.0+update67_amd64.deb
  • Update your java links to point at this new Oracle Java package. To find the name of this package take a look in /usr/lib/jvm/. There should be a directory named j2sdk1.7-oracle (or something similar). Use this directory name in a call to update-java-alternatives.
    # update-java-alternatives -s j2sdk1.7-oracle
  • Test your system is using the correct version of Java using,
    $ java -version
     java version "1.7.0_67"
     Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
     Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
    $ javac -version
     javac 1.7.0_67

Install the Android SDK Tools

The Android SDK Tools package contains the SDK Manager. You’ll need this to download one or more SDKs.

Download “SDK Tools Only” from https://developer.android.com/sdk/index.html. Extract the downloaded file to your applications directory, e.g. ~/apps/.

Install Android Studio

Download Android Studio from http://developer.android.com/sdk/installing/studio.html.

Unzip the downloaded file to whereever you keep your apps, e.g. ~/apps/.

To start Android Studio run <install dir>/android-studio/bin/studio.sh.

Configure the location of your Android SDK by selecting “Configure” -> “Project Defaults” -> “Project Structure” -> “Android SDK Location”. Enter the directory where you extracted the Android SDK Tools.

I find it useful to create a launcher icon. This way I can launch Android Studio by clicking Alt-F1 and entering the first few characters of the app name. To create a launcher icon from the Welcome splash screen select “Configure” -> “Create Desktop Entry”.

References

Installing Debian 7.1 “Wheezy”

Every time I install Linux, no matter what the distribution, I have the same problems; configuring video and audio. When I installed Debian 7.1 recently I documented the steps I took. Hopefully you’ll find something of use here.

My Hardware

Video: Advanced Micro Devices [AMD] nee ATI RV710 [Radeon HD 4350]
Sound: Advanced Micro Devices [AMD] nee ATI RV710/730 HDMI Audio [Radeon HD 4000 series]
Monitor: Dell SP2208WFP (with built in webcam)
WebCam: OmniVision Technologies, Inc. Monitor Webcam

Video and sound devices can be determined using lspci. The Webcam is a USB device so it can be determined using lsusb.

Video

For regular use the opensource Radeon display driver is considered adequate. If you’re a gamer or just want to squeeze every bit of performance out of your card then you might consider the AMD proprietary display driver. For my needs the opensource driver is fine.

On the first boot GNOME 3 failed to load. The installer recognized my video card and installed the correct package, xserver-xorg-video-radeon, but as documented on the wiki my card requires proprietary firmware. This firmware is available in non-free. The steps to install it are,

  1. Open up Synaptic Package Manager and select Settings, Repositories from the menu
  2. Tick the checkbox next to Non-DFSG-compatible Software (non-free)
  3. Click Close and then the Reload toolbar button
  4. Search for and install firmware-linux-nonfree
  5. Reboot

GNOME 3 should start successfully after the reboot.

Audio

When I didn’t hear any sound I thought I was going to be in for a world of pain with sound drivers, ALSA and PulseAudio. Luckily all I needed to do was select the right output device. My system has two sound devices, an onboard Intel device and the sound device on the AMD video card. My monitor is connected by HDMI. I have a soundbar on my monitor drawing it’s signal from the HDMI cable. All I needed to do was,

  1. Open up the System Settings
  2. Click the Sound icon
  3. In the Output tab select the HDMI audio device, i.e. RV710/730 HDMI Audio [Radeon HD 4000 series] Digital Stereo (HDMI)

Update (Aug-2014): HDMI audio is disabled by default in the kernel/audio driver. To enable it,

  1. Edit /etc/default/grub and append “radeon.audio=1” to the GRUB_CMDLINE_LINUX_DEFAULT variable.
  2. Open a terminal execute sudo update-grub
  3. Reboot

Source: No sound on HDMI with Radeon driver

Webcam

My monitor has a built in Webcam. It’s an OmniVision Technologies, Inc. Monitor Webcam. I managed to get it working but it’s a bit hit and miss. When then uvcvideo kernel module is loaded it should create the device /dev/video0. Sometimes it does, sometimes it doesn’t. To get it to work is a matter of removing the module and reloading. After two or three attempts it usually works. Not very satisfying but I don’t use it very often so I’m not that bothered.

$ sudo modprobe -r uvcvideo     # unload the module
$ sudo modprobe uvcvideo        # load the module

Once the module loads successfully and creates the /dev/video0 device you can use the Cheese application to test it.

Microphone

The mic didn’t appear to be working initially but as with audio out all I needed to do was select the right device and turn up the volume. I use the mic on my monitor (attached to the webcam).

  1. Open up the System Settings
  2. Click the Sound icon
  3. Select the Input tab
  4. Select the Monitor Webcam Analog Stero device
  5. Turn the volume up to 100%

After getting the hardware working these are the applications I installed.

Flash Player

The Flash Player package is avaiable from contrib.

  1. Open up Synaptic Package Manager and select Settings, Repositories from the menu
  2. Tick the checkbox next to DFSG-compatible Software with Non-Free Dependencies (contrib)
  3. Click Close and then the Reload toolbar button
  4. Search for and install flashplugin-nonfree
  5. Restart your browser if you

Reference: FlashPlayer on the Debian Wiki

Java Browser Plugin

Simply install the package icedtea-6-plugin.

gThumb

gThumb is a photo manager. I prefer it over the default, Shotwell, because it feels quicker and more lightweight. I already organise my photos in a year, month, day folder structure so I have no need for anything fancy.

While gThumb itself is very easy to install (the gthumb package) it was failing to properly list all the photos and videos in a directory containing one or more videos. The problem was it couldn’t find the image it uses to surround videos (to give the reel effect). The fix is simple,

$ sudo ln -s /usr/share/gthumb/ui/filmholes.png /usr/share/filmholes.png

Reference: Redhat Bug 842183 – Gthumb breaks file list on movies

Android Tools

The Android Tools are used when developing Android applications. After installing the Android SDK I found adb wouldn’t work. When I ran it it gave a “file not found”. The problem is down to missing shared libraries. Here are the steps I took to fix it,

$ sudo dpkg --add-architecture i386
$ sudo aptitude update
$ sudo aptitude install libstdc++6:i386 libgcc1:i386 zlib1g:i386 libncurses5:i386

Reference: Installing Android SDK tools on Debian Wheezy