App Engine’s URL Fetch API supports fetching URLs asynchronously. However, a request handler may only simultaneously fetch up to 10 URLs. To fetch more than 10, it must wait for one to finish before starting another. This is a little tricky to do efficiently*, so I put together a Python module which takes care of the details. The module provides an AsyncURLFetchManager class with a simple interface – just tell it what URLs you want and it fetches them as quickly as possible. This interface also simplifies the starting of an asynchronous request into a single method call:
fetch_asynchronously(url)
You can also pass fetch_asynchronously() any arguments which urlfetch.make_fetch_call() accepts (e.g., method, payload). You can also ask it for a callback which will conveniently include the RPC object (which contains the results) as well as any other positional or keyword arguments you would like.
At the end of your request, just call wait() to ensures that any pending fetches and their callbacks are completed prior to the request handler terminating.
* Unfortunately, App Engine does not currently provide select() or any other non-blocking mechanism which can check if an RPC has completed. Once it does, this implementation could be improved to ensure that it only waits on an RPC which has already completed (currently we just have to wait on the oldest one – this is sub-optimal since later RPCs may actually finish first).
David Underhill Google App Engine asynchronous, Google App Engine, Python, urlfetch
CraigNotes is a free service which helps you track your favorite ads on Craigslist.
Motivation. I love Craigslist – I’ve used it to find great housing options when moving to a new area, as well as deals for various appliances and furniture. Unfortunately, I always find myself awkwardly scratching notes in a spreadsheet as I try to track the most promising ads. This cumbersome process inspired CraigNotes.
Features. CraigNotes is a simple website which presents Craigslist ads inside an enhanced interface. CraigNotes lets you rate ads, take notes, and hide irrelevant spam. It automatically fetches the latest ads which match your search, and lets you continue to view old ads – even after they disappear from Craigslist (which only keeps them around for a week).
This simple functionality makes it a lot easier to keep your favorites at your fingertips, along with any additional notes you want to remember (I jot down details I receive from the ad’s poster when I contact them, as well as my opinion about the ad).
Give CraigNotes a try and let me know what you think!
CraigNotes Screenshot:
David Underhill Software CraigNotes, Craigslist, Google App Engine, interface
You may have some functionality on your app engine site that you want to protect from robots and prevent users from executing too frequently. For example, perhaps users can leave comments but you only want them to be able to leave a comment every N seconds – faster than that and the “user” is either a bot or is not using the system as intended.
One way to discourage this behavior is to limit how often a user can take a certain action to a fixed rate. I’ve created a RateLimiter class which handles the logic of tracking how quickly a user is making requests, and determines when your code (optionally) should challenge the user with a captcha before allowing them to continue. If you simply want to rate limit the user’s requests, you can ignore the captcha business and just return an error to the user whenever they exceed the allowed rate.
The source is available at http://gist.github.com/437051 (including the optional captcha handling code).
Example Usage:
The example code below shows a rate limiter which allows a user to interact with a particular page once every 2 seconds. It also gives the user 3 “tokens” which allows the user to violate this limit by up to 3 requests. Tokens are consumed if a user makes a request within 2 seconds of the previous request. Tokens are returned if the user if the user slows down, or if the user solves a captcha.
This example is written as if the request is expected to be made via JavaScript on your page. The client-side JavaScript would check the response for the 'captcha-show' text and prompt the user with a captcha if that test was present. When the captcha is answered, another AJAX call would be made to send the user’s response to the CaptchaHandler class in rate_limit.py. You are free to integrate the captcha challenge however you like. Just call RateLimiter.captcha_solved() or RateLimiter.rate_limit(uid, captcha_solved=True) when the user meets your challenge (it doesn’t even have to be a captcha).
David Underhill Google App Engine bot, captcha, Google App Engine, Python, rate limit