<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>David Underhill &#187; preprocess</title>
	<atom:link href="http://dound.com/tag/preprocess/feed/" rel="self" type="application/rss+xml" />
	<link>http://dound.com</link>
	<description>dound&#039;s space on the web</description>
	<lastBuildDate>Tue, 31 Jan 2012 10:57:50 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.1.3</generator>
		<item>
		<title>Python logging and performance: how to have your cake and eat it too</title>
		<link>http://dound.com/2010/02/python-logging-performance/</link>
		<comments>http://dound.com/2010/02/python-logging-performance/#comments</comments>
		<pubDate>Sun, 07 Feb 2010 05:08:48 +0000</pubDate>
		<dc:creator>David Underhill</dc:creator>
				<category><![CDATA[Coding]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[comment]]></category>
		<category><![CDATA[logging]]></category>
		<category><![CDATA[performance]]></category>
		<category><![CDATA[preprocess]]></category>
		<category><![CDATA[script]]></category>

		<guid isPermaLink="false">http://dound.com/?p=268</guid>
		<description><![CDATA[I love Python&#8216;s logging module. I use it all the time to log a wide variety of information &#8212; messages to help me debug as well as informative messages for the user. Though you can toggle which messages you want to be printed, if the Python interpreter encounters a logging method call it still creates [...]]]></description>
			<content:encoded><![CDATA[<p>I love <a href="http://www.python.org">Python</a>&#8216;s <a href="http://docs.python.org/library/logging.html">logging module</a>.  I use it all the time to log a wide variety of information &#8212; messages to help me debug as well as informative messages for the user.  Though you can toggle which messages you want to be printed, if the Python interpreter encounters a logging method call it still creates the string for the log message (the argument to the method) (sadly there Python doesn&#8217;t have <a href="http://www.haskell.org/haskellwiki/Performance/Laziness">lazy evaluation</a> like <a href="http://www.haskell.org/">Haskell</a>).  If creating this string is expensive, then your application&#8217;s performance may suffer.  Unfortunately, there is no Python preprocessor (like C&#8217;s <a href="http://gcc.gnu.org/onlinedocs/cpp/">cpp</a> &#8230; though <a href="http://code.google.com/p/preprocess/">preprocess</a> might be able to do it) so it is difficult to automatically remove a large number of logging statements prior to running an application in a production environment.</p>
<p>The best solution I&#8217;ve seen is to prefix logging statements with <code>if __debug__:</code> so that they are optimized away by <code>python -O</code> (see <a href="http://stackoverflow.com/questions/2006190/python-equivalent-of-define-func-or-how-to-comment-out-a-function-call-in-p">this post on StackOverflow</a>).  I like it, but it unfortunately requires this statement to be prefixed to every logging statement I don&#8217;t want in a production environment.  That&#8217;s a lot of ugly extra code and it isn&#8217;t easy to change which statements it applies to either.</p>
<p>I decided to write a script which automatically parses a Python file and replaces logging statements of a particular level with a <code>pass</code> statement and a commented out copy of the logging code.  It can also do the reverse operation.  It has some limitations (see the code, or run the script with the <code>--help</code> option), but it should work for most Python files.  I used it for the <a href="http://yuba.stanford.edu/vns">VNS</a> project and it successfully operated on every file in the project.  It also improved performance dramatically &#8211; the maximum throughput of the VNS simulator <a href="http://yuba.stanford.edu/vns/2010/02/fairness/">increased by 25%</a>!  In comparison, running the code with <a href="http://psyco.sourceforge.net/">Psyco</a> only garnered a 6% improvement (though pretty substantial for the minimal <a href="http://github.com/dound/vns/commit/54d5415a43043062ae6195b828707692b9231aab">13 lines</a> I had to add to take advantage of it).</p>
<p>I think this script is worth using before running your code in a production environment if you are a heavy user of the logging module like I am.  You can find the code <a href="http://www.siafoo.net/snippet/348">here</a> (it is hosted on <a href="http://www.siafoo.com">Siafoo</a>, a neat site for sharing code).  Here&#8217;s the latest version of the code:<br />
<script type='text/javascript' src='http://www.siafoo.net/snippet/348/embed.js?nolinenos'></script></p>
]]></content:encoded>
			<wfw:commentRss>http://dound.com/2010/02/python-logging-performance/feed/</wfw:commentRss>
		<slash:comments>3</slash:comments>
		</item>
	</channel>
</rss>

