David Underhill » Python

PostgreSQL UPSERT (in Python)

David Underhill — Mon, 10 Jan 2011 00:53:25 +0000

PostgreSQL does not yet support the UPSERT command (though it is on their Todo list). If you have a row you want to update (if it already exists in the database) or insert (if it doesn’t exist yet), then PostgreSQL unfortunately makes you implement the logic yourself. Other popular databases like SQLite (INSERT OR IGNORE) and MySQL (ON DUPLICATE KEY UPDATE) both support upserting. I haven’t run across a generic PL/pgSQL function which can do this, but you could write a trigger (like this one) for each table where this functionality is needed.

Unfortunately, this is a bit of a pain if you want to use UPSERT on many tables, so I wrote a Python method which takes care of the UPSERT logic generically. To use it, you call it with a cursor connected to your database, the schema and table name, a list of primary key field names, and the key-value pairs for each field.

For example, let’s say you have a table which tracks scores (and only the last score counts):

CREATE TABLE MySchema.Scores (
    user_id integer PRIMARY KEY,
    score   integer NOT NULL
);

To UPSERT a row into this table you would:

db_conn = psycopg2.connect("...")
db_cur = db_conn.cursor()
upsert(db_cur, 'Scores', ('user_id',), schema='MySchema', user_id=..., score=...)
db_conn.commit()

Here’s the code for the Python-based upsert method:

Python logging and performance: how to have your cake and eat it too

David Underhill — Sun, 07 Feb 2010 05:08:48 +0000

I love Python‘s logging module. I use it all the time to log a wide variety of information — messages to help me debug as well as informative messages for the user. Though you can toggle which messages you want to be printed, if the Python interpreter encounters a logging method call it still creates the string for the log message (the argument to the method) (sadly there Python doesn’t have lazy evaluation like Haskell). If creating this string is expensive, then your application’s performance may suffer. Unfortunately, there is no Python preprocessor (like C’s cpp … though preprocess might be able to do it) so it is difficult to automatically remove a large number of logging statements prior to running an application in a production environment.

The best solution I’ve seen is to prefix logging statements with if __debug__: so that they are optimized away by python -O (see this post on StackOverflow). I like it, but it unfortunately requires this statement to be prefixed to every logging statement I don’t want in a production environment. That’s a lot of ugly extra code and it isn’t easy to change which statements it applies to either.

I decided to write a script which automatically parses a Python file and replaces logging statements of a particular level with a pass statement and a commented out copy of the logging code. It can also do the reverse operation. It has some limitations (see the code, or run the script with the --help option), but it should work for most Python files. I used it for the VNS project and it successfully operated on every file in the project. It also improved performance dramatically – the maximum throughput of the VNS simulator increased by 25%! In comparison, running the code with Psyco only garnered a 6% improvement (though pretty substantial for the minimal 13 lines I had to add to take advantage of it).

I think this script is worth using before running your code in a production environment if you are a heavy user of the logging module like I am. You can find the code here (it is hosted on Siafoo, a neat site for sharing code). Here’s the latest version of the code:

Integrating Twisted with a pcap-based Python packet sniffer

David Underhill — Wed, 09 Sep 2009 00:53:34 +0000

Twisted is an awesome event-driven networking engine. Unfortunately, it does not have good support for interfacing with raw sockets (unlike its support for many network protocols, which is amazing). Anyway, I recently needed to work with raw sockets so I had to find a way to make it work with Twisted. Though Twisted does have a module (twisted.pair) which tries to provide some support for raw sockets, the module is poorly documented and requires a library which is not readily available.

Luckily, I stumbled on a module which works on top of the libpcap packet capture library called pcapy. It is simple to use, and thread-safe — and easy to integrate into a Twisted-based project.

I put together a short sample (see below) which shows how to capture raw packets alongside the main Twisted event loop. It would be trivial to extend this example to also write to a raw socket (using an ordinary Python socket). This example can also be downloaded here.

# This sample shows how to run a libpcap-based packet sniffer concurrently with
# the Twisted framework.  The Twisted component is an "Echo" TCP server
# (listening on port 9999) which prints everything it receives.  When a client
# connects, it starts the pcap thread.  When the pcap thread receives a packet,
# it sends a message to the client telling it the size of the received packet.
# Finally, when the client disconnects the program is terminated.
 
# To try this contrived example out, run this script as root (so that it can use
# pcap) and then connect to the echo server (e.g., telnet localhost 9999).  Note
# that the pcap parameters are hard-coded.  This code uses twisted 8.0.2 and
# pcapy-0.10.4.
 
import os
 
from pcapy import open_live
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
 
# pcap settings
DEV          = 'eth0'  # interface to listen on
MAX_LEN      = 1514    # max size of packet to capture
PROMISCUOUS  = 1       # promiscuous mode?
READ_TIMEOUT = 100     # in milliseconds
PCAP_FILTER  = ''      # empty => get everything (or we could use a BPF filter)
MAX_PKTS     = -1      # number of packets to capture; -1 => no limit
 
def run_pcap(f):
    # the method which will be called when a packet is captured
    def ph(hdr, data):
        print 'pcap heard: when=%s sz=%dB' % (hdr.getts(), len(data))
        # thread safety: call from the main twisted event loop
        reactor.callFromThread(f, len(data))
 
    # start the packet capture
    p = open_live(DEV, MAX_LEN, PROMISCUOUS, READ_TIMEOUT)
    p.setfilter(PCAP_FILTER)
    print "Listening on %s: net=%s, mask=%s" % (DEV, p.getnet(), p.getmask())
    p.loop(MAX_PKTS, ph)
 
# a silly echo server which prints what it receives and sends info about the
# size of each packet captured on DEV
class Echo(Protocol):
    def connectionLost(self, reason):
        os._exit(0) # kill the whole process
 
    def connectionMade(self):
        # run pcap in another thread (it will run forever)
        reactor.callInThread(run_pcap, self.pcapDataReceived)
 
    def dataReceived(self, data):
        print 'echo got: %s' % data
 
    def pcapDataReceived(self, sz):
        self.transport.write('pcap got: %uB\n' % sz)
 
# starts the silly echo server on port 9999
def main():
    factory = Factory()
    factory.protocol = Echo
    reactor.listenTCP(9999, factory)
    reactor.run()
 
if __name__ == "__main__":
    main()

Python + Twisted Length-Type-Based Protocol Client / Server

David Underhill — Sun, 08 Mar 2009 09:25:12 +0000

It seems like I often have a need to work with a simple TCP protocol whose messages have a header which starts with the length of the message and an integer representing the message type (OpenFlow is one of many such protocols). To save myself the trouble of creating and debugging a very similar custom implementation each time I have this need, I decided to package it as a simple Python framework which does this for me. It is based on the event-driven Twised networking engine. Using this simple extension on top of Twisted has a number of benefits:

Automatic handling of the length and type fields when sending and receiving messages.
Automatic unpacking of messages based on type.
Client automatically tries to reconnect if the connection is lost.
Server can handle any number of clients simultaneously.

You can view the official package on the PyPi website here. My local page for the package is here — please view it for an example on how to use this package.