tomdyson.org

music, code, photos etc.

April 3, 2013

March 25, 2013

February 10, 2013

November 12, 2012

November 2, 2010

how smories uses redis

One of the things I’m working on outside Torchbox is Smories, a site where children tell stories to other children. Check it out! The stories are beautifully read, and the collection is growing fast.

Smories is a Django app. Although the little server it runs on handles the spiky traffic fine I’ve been fiddling with Redis to prepare the site for when it eventually overtakes Facebook. I love Redis. It feels like a natural, ultra-performant extension to Python’s easy way of handling data types. Here are three ways Smories uses it:

  • Cached queries - of course Django’s cache framework makes this trivial and the standard approach is Memcached for the backend. If your app is already using Redis you can just swap in one of the Redis cache backends for Django (I use this one) but if the value you want to cache is a collection – like the results of a database query – it’s cleaner to retain its manipulable structure in the cache, rather than a serialised blob. For example, to pick a random Smory from the cached query I can call SRANDMEMBER directly on the set, whereas in the traditional Memcached model I’d retrieve the key, flatten the deserialised resultset into a list, pull out a random item and discard the rest.
  • Rate limiting - We run competitions for authors who submit their stories, with financial prizes for the Smories with the most views. Some unscrupulous authors and their mothers watch their own videos hundreds of times a day in the hope of increasing their views. For each Smory view I hash a collection of details about the browser together with the video URL and use this as an expiring Redis key. If a subsequent request matches the key, I’ll increment the value. If it matches, and the value is too high, I ignore the view and reset the expiry. You could use Memcached in just the same way, but you probably shouldn’t; Memcached’s intentional lack of durability means you’d lose the vote throttling state when you restart the daemon (e.g. to tweak its memory allocation).
  • Activity tracking - Ralph, my collaborator on Smories, is a data addict. I built him a simple realtime activity tracker as a little present. It uses the excellent Pusher service for websocket infrastructure; relevant Django views call a notification function which wraps the Python Pusher library to broadcast the event to connected clients. At the same time, new events are pushed on to a stack representing recent activity, so on Ralph’s first visit he sees the last ten events before they’re quickly replaced with the activities taking place while he watches. Here’s how it looks right now. Redis lists are well-suited for this kind of capped collection: inserts are practically free, and the implementation (push then trim) is simple and readable.