David Underhill

I love Python’s logging module. I use it all the time to log a wide variety of information — messages to help me debug as well as informative messages for the user. Though you can toggle which messages you want to be printed, if the Python interpreter encounters a logging method call it still creates the string for the log message (the argument to the method) (sadly there Python doesn’t have lazy evaluation like Haskell). If creating this string is expensive, then your application’s performance may suffer. Unfortunately, there is no Python preprocessor (like C’s cpp … though preprocess might be able to do it) so it is difficult to automatically remove a large number of logging statements prior to running an application in a production environment.

The best solution I’ve seen is to prefix logging statements with if __debug__: so that they are optimized away by python -O (see this post on StackOverflow). I like it, but it unfortunately requires this statement to be prefixed to every logging statement I don’t want in a production environment. That’s a lot of ugly extra code and it isn’t easy to change which statements it applies to either.

I decided to write a script which automatically parses a Python file and replaces logging statements of a particular level with a pass statement and a commented out copy of the logging code. It can also do the reverse operation. It has some limitations (see the code, or run the script with the --help option), but it should work for most Python files. I used it for the VNS project and it successfully operated on every file in the project. It also improved performance dramatically – the maximum throughput of the VNS simulator increased by 25%! In comparison, running the code with Psyco only garnered a 6% improvement (though pretty substantial for the minimal 13 lines I had to add to take advantage of it).

I think this script is worth using before running your code in a production environment if you are a heavy user of the logging module like I am. You can find the code here (it is hosted on Siafoo, a neat site for sharing code). Here’s the latest version of the code:

David Underhill Coding, Python comment, logging, performance, preprocess, Python, script

A little while ago I picked up a 26″ monitor (NEC MultiSync LCD2690WUXi). Unfortunately, I found that when I connected the monitor to my Ubuntu Linux box that I could only use up to 1280×1028 — even though the monitor’s native resolution was 1920×1080! I also had this problem on my Windows and SuSE machines, so I suspect the monitor is not properly reporting its maximum resolution via EDID.

I used the command-line utility xrandr to fix the problem. Running the tool with no arguments prints a list of displays and available display modes for each. This is handy since you need the name assigned to your display by your OS for the next step. Next, use the “–newmode” option with xrandr and specify the modeline which describes the display configuration you wish to use. This modeline generator might help you create the modeline you need. Once you create the new mode, use the “–addmode” option to add it to the list of modes supported by your monitor.

Finally, add this command to your ~/.xprofile file (or something similar) so that when you start your machine the new mode is automatically added and available (this way Ubuntu automatically reselects it too). This is what I ended up adding to my ~/.xprofile file:

xrandr --newmode "1920x1200_50Hz" 128,300 1920 1968 2000 2079 1200 1203 1209 1234 +hsync -vsync
xrandr --addmode HDMI-0 "1920x1200_50Hz"

Note: If you set your refresh rate too high, your monitor will probably flicker occasionally. If this happens, try lowering the refresh rate by lowering the pixel clock value (the first number in the modeline).

David Underhill Linux EDID, Linux, modeline, monitor, resolution, X11, xrandr

A little while ago I bought Adobe Creative Suite CS4 Design Standard so that I could use Photoshop and Acrobat Pro on my Windows 7 machine (which I use for photo editing and organizing). Installing, registering, and activating the 64-bit version was painless, except for one problem: whenever I launched Acrobat, it would work fine for a few seconds until I was interrupted by an annoying dialog box which stated: “Adobe Acrobat was installed as part of a suite. To enable Adobe Acrobat, please start another component of this suite (such as Adobe Photoshop).” After clicking “Ok” Acrobat would shutdown.

Needless to say, this made the Acrobat part of my installation useless. Strangely, the dialog persisted even if I was already running Photoshop when I started Acrobat. I tried deactivating and reactivating, reinstalling, and so forth — all to no avail.

Google didn’t seem to have any advice on the matter, so I gave in and tried calling technical support. Though the representative was nice, he unfortunately didn’t have an immediate solution to the problem. I kept playing around with my installation and stumbled across something that worked — instead of running the 64-bit version of Photoshop, I ran the ordinary version (it is installed in your start menu alongside the 64-bit version). Ever since then Acrobat has worked fine (even when it is the only Adobe product running)!

It seems like the 64-bit version of Photoshop doesn’t properly communicate the activation details to Acrobat — a pretty bizarre (and annoying) bug for an Adobe product (which I am usually very fond of). Thankfully, the workaround is pretty simple — just run the regular version of Photoshop (presumably running Illustrator or any of the other programs in the suite would also do the trick).

David Underhill Software Acrobat, Adobe, bug, workaround

Twisted is an awesome event-driven networking engine. Unfortunately, it does not have good support for interfacing with raw sockets (unlike its support for many network protocols, which is amazing). Anyway, I recently needed to work with raw sockets so I had to find a way to make it work with Twisted. Though Twisted does have a module (twisted.pair) which tries to provide some support for raw sockets, the module is poorly documented and requires a library which is not readily available.

Luckily, I stumbled on a module which works on top of the libpcap packet capture library called pcapy. It is simple to use, and thread-safe — and easy to integrate into a Twisted-based project.

I put together a short sample (see below) which shows how to capture raw packets alongside the main Twisted event loop. It would be trivial to extend this example to also write to a raw socket (using an ordinary Python socket). This example can also be downloaded here.

# This sample shows how to run a libpcap-based packet sniffer concurrently with
# the Twisted framework.  The Twisted component is an "Echo" TCP server
# (listening on port 9999) which prints everything it receives.  When a client
# connects, it starts the pcap thread.  When the pcap thread receives a packet,
# it sends a message to the client telling it the size of the received packet.
# Finally, when the client disconnects the program is terminated.
 
# To try this contrived example out, run this script as root (so that it can use
# pcap) and then connect to the echo server (e.g., telnet localhost 9999).  Note
# that the pcap parameters are hard-coded.  This code uses twisted 8.0.2 and
# pcapy-0.10.4.
 
import os
 
from pcapy import open_live
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
 
# pcap settings
DEV          = 'eth0'  # interface to listen on
MAX_LEN      = 1514    # max size of packet to capture
PROMISCUOUS  = 1       # promiscuous mode?
READ_TIMEOUT = 100     # in milliseconds
PCAP_FILTER  = ''      # empty => get everything (or we could use a BPF filter)
MAX_PKTS     = -1      # number of packets to capture; -1 => no limit
 
def run_pcap(f):
    # the method which will be called when a packet is captured
    def ph(hdr, data):
        print 'pcap heard: when=%s sz=%dB' % (hdr.getts(), len(data))
        # thread safety: call from the main twisted event loop
        reactor.callFromThread(f, len(data))
 
    # start the packet capture
    p = open_live(DEV, MAX_LEN, PROMISCUOUS, READ_TIMEOUT)
    p.setfilter(PCAP_FILTER)
    print "Listening on %s: net=%s, mask=%s" % (DEV, p.getnet(), p.getmask())
    p.loop(MAX_PKTS, ph)
 
# a silly echo server which prints what it receives and sends info about the
# size of each packet captured on DEV
class Echo(Protocol):
    def connectionLost(self, reason):
        os._exit(0) # kill the whole process
 
    def connectionMade(self):
        # run pcap in another thread (it will run forever)
        reactor.callInThread(run_pcap, self.pcapDataReceived)
 
    def dataReceived(self, data):
        print 'echo got: %s' % data
 
    def pcapDataReceived(self, sz):
        self.transport.write('pcap got: %uB\n' % sz)
 
# starts the silly echo server on port 9999
def main():
    factory = Factory()
    factory.protocol = Echo
    reactor.listenTCP(9999, factory)
    reactor.run()
 
if __name__ == "__main__":
    main()

David Underhill Coding, Python pcap, pcapy, Python, raw socket, sniffer, twisted

For the second year in a row, the OpenFlow team has captured the Best Demo award at SIGCOMM 2009. Read the post on the OpenFlow blog for more details! Though I only focused on one of the interfaces, all four of the OpenFlow demos built their GUIs on the ENVI framework which I created with the OpenFlow team!

David Underhill OpenFlow

I recently picked up a 26″ monitor (NEC MultiSync LCD2690WUXi – great price, even new, and excellent color gamut). Unfortunately, I found that when I connected to the monitor in Vista (which I use when I want to run Adobe products) that I could only use up to 1280×1028 — even though the monitor’s native resolution was 1920×1080! It took a lot of searching, trial, and error to piece together a solution, but the following steps should enable you to run your monitor at full resolution if you have an ATI video card:

Update to the latest ATI driver and Catalyst Control Center (optional) – This should be straightforward if you own a desktop, but on my Dell laptop ATI’s installers won’t upgrade your drivers (or even install new ones after you’ve uninstalled the old ones — apparently only the OEM drivers will install). Anyway, Ruud Ketelaars has developed “Mobility Modder” which can tweak the installation files so that you can upgrade your laptop’s ATI drivers. You can find his software and instructions here (scroll down).

Add the max resolution of your monitor to the available options:

1. Open ATI’s Catalyst Control Center (CCC) by right-clicking on your desktop and clicking on the “Catalyst Control Center” option.
2. Make sure you are in advanced mode (lots of menus on the left side).
3. One of the menu options on the left side will be something like “Digital Panel (DVI) 2″ and will probably be listed just under “Notebook Panel Properties 0″ if you are using a laptop. Click on the “HDTV Support” submenu of the “Digital Panel” menu.
4. Under “HDTV Support”, click the checkbox next to the resolution and refresh rate which you want to enable for your monitor. For my monitor, this was “1920×1080p50″ (50Hz).
5. Click “Ok.”
6. Open Display Settings like usual and you’ll be able to choose the new resolutions you enabled (right-click on the desktop, choose Personalize, then Display Settings)

David Underhill Hardware ATI, driver, monitor, resolution, vista

I recently wanted to put a footnote reference inside a table. Unfortunately, LaTeX makes it somewhat difficult to add footnotes inside floats (e.g., tabular). If you try to put a footnote inside a tabular, then pdflatex will show the reference but not the footnote itself! I came across several suggestions for fixing this:

One idea is to put the table in a minipage. This causes the footnote to show up at the bottom of the table (in its own numbering system) — but I wanted the footnote to show up at the bottom of the page like other footnotes!

Another idea was to manually specify the footnote number inside the text and then use the \footnotetext command (outside the tabular) to manually add the footer. Unfortunately, this is not a robust solution since it forces you to manually maintain this footnote number inside the tabular.

Building on the previous idea, I discovered a way to make footnotes appear inside tabulars without breaking the automatic numbering of footnotes. Here is my approach:

1. Include the “fmtcount” package so that you can display the values of counters (e.g., the footnote counter):

   \usepackage{fmtcount}

2. Immediately before your tabular, increment the footnote counter:

   \addtocounter{footnote}{1}

3. Next, specify the contents of the footnote:

   \footnotetext[\value{footnote}]{your text here}

4. Finally, add a reference to the footnote inside the table:

   $^{\decimal{footnote}}$

You can extend this idea to add multiple footnotes within a single tabular by adjusting the counters (using \addtocounter) appropriately. Here is a complete example of how to add two footnotes inside a single tabular (you can see the PDF output here):

\documentclass[12pt]{article}
 
\usepackage{fmtcount} % displaying latex counters
 
\begin{document}
    \title{An Example of Footnotes Inside a Tabular}
    \author{David Gridley Underhill}
    \maketitle
 
% manually add a footnote which exists inside the table
\addtocounter{footnote}{1}
\footnotetext[\value{footnote}]{my first footnote}
 
% add another footnote
\addtocounter{footnote}{1}
\footnotetext[\value{footnote}]{my second footnote}
 
% reset the counter to the first footnote's value
\addtocounter{footnote}{-1}
 
\begin{tabular}{|l|l|}
  \hline
  % this next row references the first footnote I added above, and then
  % advances the counter to the next footnote.
  {\bf First Column} & {\bf Second Column}$^{\decimal{footnote}}$\addtocounter{footnote}{1} \\
 
  \hline
  % now reference the second footnote from above -- don't increment the footnote 
  % counter beyond the last footnote!
  X & Y$^{\decimal{footnote}}$ \\
 
  \hline
\end{tabular}
 
\end{document}

David Underhill Coding, LaTeX caption, figure, float, footnote, latex, table, tabular

I’ve been looking for a way to have github, my favorite repository hosting service, send emails to interested parties whenever someone pushes new commits into the repository, They don’t seem to provide this service directly, but they do provide both an API for programatically querying the site as well as post-receive callbacks. The latter sends a POST request to URL(s) of your choice. The POST request includes JSON data which contains information about the repository and the new commits (for details, see here). Using this, I was able to put together a relatively simple PHP script which repackages this information into a human-readable form and sends it off in an email (source code here).

I wanted to use the github API to get a list of those who watch the project, and then send an email to those people. Unfortunately the API does not let you query that yet. Instead, my script lets you specify the recipients manually. Alternatively, a simple mailing list implementation is provided so people can sign up for the post-receive emails through a webpage instead.

The only downside is that the JSON github provides does not include information about how many lines were modified, just files. In this respect, Andy Parkin’s post-receive script produces slightly more informative e-mails (but of course it doesn’t work with github).

David Underhill Coding, PHP, version control git, github, JSON, post-receive

I’ve revamped the front page to list current projects instead of past ones. I’ve also updated the Projects page to include some more recent projects including jToolbar, a minimum spanning tree algorithm library, and the ENVI network visualization and control framework. I also updated ltprotocol to provide additional and more symmetric (between the client and server) callbacks.

David Underhill Coding, Software project, update

I recently had a need to rewrite a git repository’s history. This isn’t generally a very good idea, though it is useful if your repository contains files it should not (such as unneeded large binary files or copyrighted material). I also am using it because I had a branch where I only wanted to merge a subset of files back into master (though there are probably better ways of doing this). Anyway, it is not very hard to rewrite history thanks to the excellent git-filter-branch tool which comes with git. However, if your goal was to reduce a large repository’s size then git-filter-branch does not quite finish the job since it makes temporary backups of the filtered out files. To remove those, you need to do a little more work. To make it easier to permanently remove files, I wrapped it in a little bash script git-remove-history (also shown below) — simply go to the root of your repository and run the script with the list of files you want to delete and it will do the rest. There is an interesting thread about doing this here on KernelTrap.

#!/bin/bash
set -o errexit
 
# Author: David Underhill
# Script to permanently delete files/folders from your git repository.  To use 
# it, cd to your repository's root and then run the script with a list of paths
# you want to delete, e.g., git-delete-history path1 path2
 
if [ $# -eq 0 ]; then
    exit 0are still
fi
 
# make sure we're at the root of git repo
if [ ! -d .git ]; then
    echo "Error: must run this script from the root of a git repository"
    exit 1
fi
 
# remove all paths passed as arguments from the history of the repo
files=$@
git filter-branch --index-filter "git rm -rf --cached --ignore-unmatch $files" HEAD
 
# remove the temporary history git-filter-branch otherwise leaves behind for a long time
rm -rf .git/refs/original/ && git reflog expire --all &&  git gc --aggressive --prune

David Underhill Coding, bash, version control bash, binary file, git, history, remove, repository, version control