Python logging and performance: how to have your cake and eat it too
I love Python‘s logging module. I use it all the time to log a wide variety of information — messages to help me debug as well as informative messages for the user. Though you can toggle which messages you want to be printed, if the Python interpreter encounters a logging method call it still creates the string for the log message (the argument to the method) (sadly there Python doesn’t have lazy evaluation like Haskell). If creating this string is expensive, then your application’s performance may suffer. Unfortunately, there is no Python preprocessor (like C’s cpp … though preprocess might be able to do it) so it is difficult to automatically remove a large number of logging statements prior to running an application in a production environment.
The best solution I’ve seen is to prefix logging statements with if __debug__: so that they are optimized away by python -O (see this post on StackOverflow). I like it, but it unfortunately requires this statement to be prefixed to every logging statement I don’t want in a production environment. That’s a lot of ugly extra code and it isn’t easy to change which statements it applies to either.
I decided to write a script which automatically parses a Python file and replaces logging statements of a particular level with a pass statement and a commented out copy of the logging code. It can also do the reverse operation. It has some limitations (see the code, or run the script with the --help option), but it should work for most Python files. I used it for the VNS project and it successfully operated on every file in the project. It also improved performance dramatically – the maximum throughput of the VNS simulator increased by 25%! In comparison, running the code with Psyco only garnered a 6% improvement (though pretty substantial for the minimal 13 lines I had to add to take advantage of it).
I think this script is worth using before running your code in a production environment if you are a heavy user of the logging module like I am. You can find the code here (it is hosted on Siafoo, a neat site for sharing code). Here’s the latest version of the code: