One way to discourage this behavior is to limit how often a user can take a certain action to a fixed rate. I’ve created a RateLimiter class which handles the logic of tracking how quickly a user is making requests, and determines when your code (optionally) should challenge the user with a captcha before allowing them to continue. If you simply want to rate limit the user’s requests, you can ignore the captcha business and just return an error to the user whenever they exceed the allowed rate.

The source is available at http://gist.github.com/437051 (including the optional captcha handling code).

Example Usage:
The example code below shows a rate limiter which allows a user to interact with a particular page once every 2 seconds. It also gives the user 3 “tokens” which allows the user to violate this limit by up to 3 requests. Tokens are consumed if a user makes a request within 2 seconds of the previous request. Tokens are returned if the user if the user slows down, or if the user solves a captcha.

This example is written as if the request is expected to be made via JavaScript on your page. The client-side JavaScript would check the response for the 'captcha-show' text and prompt the user with a captcha if that test was present. When the captcha is answered, another AJAX call would be made to send the user’s response to the CaptchaHandler class in rate_limit.py. You are free to integrate the captcha challenge however you like. Just call RateLimiter.captcha_solved() or RateLimiter.rate_limit(uid, captcha_solved=True) when the user meets your challenge (it doesn’t even have to be a captcha).

Here is a screenshot of the UI (with fake “random” failures inserted to show how failures are handled):

Beaker is a solid implementation, but it lacks support for memcache on app engine. This means every request must go to the datastore to fetch session data – yuck.

gaeutilities is designed for app engine and takes advantage of both memcache and the datastore. Unfortunately, the code is a bit heavyweight – it is coupled to unrelated functionality (e.g., “flash” messaging) and it is complicated by support for options I do not need (e.g., cookie-only sessions and automatic token rotation). Most significantly, its performance suffers from excess API calls and inefficient model storage.

Since I was unsatisfied with these options, I wrote my own sessions middleware, gae-sessions. It strives to be lightweight, fast (but reliable), secure, and easy to use. I ended up with a pretty small library (200 lines of code) which met these goals. It uses memcache (for speed) and the datastore (for reliability) but only reads and writes when it must. db.Model objects are efficiently stored by converting them to protobufs instead of using the automatic pickling functionality (which is slow since app engine lacks cPickle).

Consider gae-sessions if you need sessions support for a Python web application hosted on Google’s app engine. The project includes demo code which you can run without modification on the app engine development server. The demo shows gae-sessions working with an OpenID-based authentication system powered by RPX (check it out to see how easy it is to integrate with RPX).

Update: I’ve created an in-depth comparison page which compares both the features and performance of alternative sessions libraries (beaker, geautilities, gmemsess, and suas) with gae-sessions.

The best solution I’ve seen is to prefix logging statements with if __debug__: so that they are optimized away by python -O (see this post on StackOverflow). I like it, but it unfortunately requires this statement to be prefixed to every logging statement I don’t want in a production environment. That’s a lot of ugly extra code and it isn’t easy to change which statements it applies to either.

I decided to write a script which automatically parses a Python file and replaces logging statements of a particular level with a pass statement and a commented out copy of the logging code. It can also do the reverse operation. It has some limitations (see the code, or run the script with the --help option), but it should work for most Python files. I used it for the VNS project and it successfully operated on every file in the project. It also improved performance dramatically – the maximum throughput of the VNS simulator increased by 25%! In comparison, running the code with Psyco only garnered a 6% improvement (though pretty substantial for the minimal 13 lines I had to add to take advantage of it).

I think this script is worth using before running your code in a production environment if you are a heavy user of the logging module like I am. You can find the code here (it is hosted on Siafoo, a neat site for sharing code). Here’s the latest version of the code:

I used the command-line utility xrandr to fix the problem. Running the tool with no arguments prints a list of displays and available display modes for each. This is handy since you need the name assigned to your display by your OS for the next step. Next, use the “–newmode” option with xrandr and specify the modeline which describes the display configuration you wish to use. This modeline generator might help you create the modeline you need. Once you create the new mode, use the “–addmode” option to add it to the list of modes supported by your monitor.

Finally, add this command to your ~/.xprofile file (or something similar) so that when you start your machine the new mode is automatically added and available (this way Ubuntu automatically reselects it too). This is what I ended up adding to my ~/.xprofile file:

xrandr --newmode "1920x1200_50Hz" 128,300 1920 1968 2000 2079 1200 1203 1209 1234 +hsync -vsync
xrandr --addmode HDMI-0 "1920x1200_50Hz"

Note: If you set your refresh rate too high, your monitor will probably flicker occasionally. If this happens, try lowering the refresh rate by lowering the pixel clock value (the first number in the modeline).

Needless to say, this made the Acrobat part of my installation useless. Strangely, the dialog persisted even if I was already running Photoshop when I started Acrobat. I tried deactivating and reactivating, reinstalling, and so forth — all to no avail.

Google didn’t seem to have any advice on the matter, so I gave in and tried calling technical support. Though the representative was nice, he unfortunately didn’t have an immediate solution to the problem. I kept playing around with my installation and stumbled across something that worked — instead of running the 64-bit version of Photoshop, I ran the ordinary version (it is installed in your start menu alongside the 64-bit version). Ever since then Acrobat has worked fine (even when it is the only Adobe product running)!

It seems like the 64-bit version of Photoshop doesn’t properly communicate the activation details to Acrobat — a pretty bizarre (and annoying) bug for an Adobe product (which I am usually very fond of). Thankfully, the workaround is pretty simple — just run the regular version of Photoshop (presumably running Illustrator or any of the other programs in the suite would also do the trick).

Luckily, I stumbled on a module which works on top of the libpcap packet capture library called pcapy. It is simple to use, and thread-safe — and easy to integrate into a Twisted-based project.

I put together a short sample (see below) which shows how to capture raw packets alongside the main Twisted event loop. It would be trivial to extend this example to also write to a raw socket (using an ordinary Python socket). This example can also be downloaded here.

# This sample shows how to run a libpcap-based packet sniffer concurrently with
# the Twisted framework.  The Twisted component is an "Echo" TCP server
# (listening on port 9999) which prints everything it receives.  When a client
# connects, it starts the pcap thread.  When the pcap thread receives a packet,
# it sends a message to the client telling it the size of the received packet.
# Finally, when the client disconnects the program is terminated.
 
# To try this contrived example out, run this script as root (so that it can use
# pcap) and then connect to the echo server (e.g., telnet localhost 9999).  Note
# that the pcap parameters are hard-coded.  This code uses twisted 8.0.2 and
# pcapy-0.10.4.
 
import os
 
from pcapy import open_live
from twisted.internet.protocol import Protocol, Factory
from twisted.internet import reactor
 
# pcap settings
DEV          = 'eth0'  # interface to listen on
MAX_LEN      = 1514    # max size of packet to capture
PROMISCUOUS  = 1       # promiscuous mode?
READ_TIMEOUT = 100     # in milliseconds
PCAP_FILTER  = ''      # empty => get everything (or we could use a BPF filter)
MAX_PKTS     = -1      # number of packets to capture; -1 => no limit
 
def run_pcap(f):
    # the method which will be called when a packet is captured
    def ph(hdr, data):
        print 'pcap heard: when=%s sz=%dB' % (hdr.getts(), len(data))
        # thread safety: call from the main twisted event loop
        reactor.callFromThread(f, len(data))
 
    # start the packet capture
    p = open_live(DEV, MAX_LEN, PROMISCUOUS, READ_TIMEOUT)
    p.setfilter(PCAP_FILTER)
    print "Listening on %s: net=%s, mask=%s" % (DEV, p.getnet(), p.getmask())
    p.loop(MAX_PKTS, ph)
 
# a silly echo server which prints what it receives and sends info about the
# size of each packet captured on DEV
class Echo(Protocol):
    def connectionLost(self, reason):
        os._exit(0) # kill the whole process
 
    def connectionMade(self):
        # run pcap in another thread (it will run forever)
        reactor.callInThread(run_pcap, self.pcapDataReceived)
 
    def dataReceived(self, data):
        print 'echo got: %s' % data
 
    def pcapDataReceived(self, sz):
        self.transport.write('pcap got: %uB\n' % sz)
 
# starts the silly echo server on port 9999
def main():
    factory = Factory()
    factory.protocol = Echo
    reactor.listenTCP(9999, factory)
    reactor.run()
 
if __name__ == "__main__":
    main()

Update to the latest ATI driver and Catalyst Control Center (optional) – This should be straightforward if you own a desktop, but on my Dell laptop ATI’s installers won’t upgrade your drivers (or even install new ones after you’ve uninstalled the old ones — apparently only the OEM drivers will install). Anyway, Ruud Ketelaars has developed “Mobility Modder” which can tweak the installation files so that you can upgrade your laptop’s ATI drivers. You can find his software and instructions here (scroll down).

Add the max resolution of your monitor to the available options:

1. Open ATI’s Catalyst Control Center (CCC) by right-clicking on your desktop and clicking on the “Catalyst Control Center” option.
2. Make sure you are in advanced mode (lots of menus on the left side).
3. One of the menu options on the left side will be something like “Digital Panel (DVI) 2″ and will probably be listed just under “Notebook Panel Properties 0″ if you are using a laptop. Click on the “HDTV Support” submenu of the “Digital Panel” menu.
4. Under “HDTV Support”, click the checkbox next to the resolution and refresh rate which you want to enable for your monitor. For my monitor, this was “1920×1080p50″ (50Hz).
5. Click “Ok.”
6. Open Display Settings like usual and you’ll be able to choose the new resolutions you enabled (right-click on the desktop, choose Personalize, then Display Settings)