Home > Coding, Python > Python logging and performance: how to have your cake and eat it too

Python logging and performance: how to have your cake and eat it too

February 6th, 2010

I love Python‘s logging module. I use it all the time to log a wide variety of information — messages to help me debug as well as informative messages for the user. Though you can toggle which messages you want to be printed, if the Python interpreter encounters a logging method call it still creates the string for the log message (the argument to the method) (sadly there Python doesn’t have lazy evaluation like Haskell). If creating this string is expensive, then your application’s performance may suffer. Unfortunately, there is no Python preprocessor (like C’s cpp … though preprocess might be able to do it) so it is difficult to automatically remove a large number of logging statements prior to running an application in a production environment.

The best solution I’ve seen is to prefix logging statements with if __debug__: so that they are optimized away by python -O (see this post on StackOverflow). I like it, but it unfortunately requires this statement to be prefixed to every logging statement I don’t want in a production environment. That’s a lot of ugly extra code and it isn’t easy to change which statements it applies to either.

I decided to write a script which automatically parses a Python file and replaces logging statements of a particular level with a pass statement and a commented out copy of the logging code. It can also do the reverse operation. It has some limitations (see the code, or run the script with the --help option), but it should work for most Python files. I used it for the VNS project and it successfully operated on every file in the project. It also improved performance dramatically – the maximum throughput of the VNS simulator increased by 25%! In comparison, running the code with Psyco only garnered a 6% improvement (though pretty substantial for the minimal 13 lines I had to add to take advantage of it).

I think this script is worth using before running your code in a production environment if you are a heavy user of the logging module like I am. You can find the code here (it is hosted on Siafoo, a neat site for sharing code). Here’s the latest version of the code:

David Underhill Coding, Python , , , , ,

  1. Wavatar
    Evan Plaice
    June 7th, 2010 at 13:45 | #1

    “there is no Python preprocessor”

    I wrote pypreprocessor to solve this issue.

    Check it out on Google code @:
    http://code.google.com/p/pypreprocessor/

  2. June 7th, 2010 at 17:46 | #2

    @Evan Plaice
    Well, there was no Python preprocessor when I wrote this post anyway :) . It looks like a very useful tool. For the specific task of removing particular logging statements, I’ll stick to the script I posted here because I don’t have to dirty my code with #ifdef’s (it is also shorter than surrounding lots of logging blocks with #ifdef’s). But for more complicated scenario I think your preprocessor tool would be handy. Thanks for sharing it!

  3. Wavatar
    koenig
  1. No trackbacks yet.