The Limits of "Free" on Google App Engine

Update: Just my luck - I write 99% of a blog post about how I got around the limitations of Google App Engine’s free tier, and then I get an email announcing how Google will drastically change their price structure not in my favour. Looks like I’m not done trying to outsmart Google’s billing department…


Google App Engine’s free tier is great for small apps, but as I discovered when Twitty City reached a gigabyte of storage, its limitations can be extremely frustrating.

Twitty City ran well for its first few months, but after 400,000 tweets its datastore was full and needed to be cleaned up. Unlike the typical MySQL/PHP setup, GAE does not give the user the ability to delete from the datastore with SQL commands: data must be obtained programmatically. The only way to delete several rows is to select them in your program’s code, iterate over them, and delete them individually. This would be tolerable, except for three factors:

  1. It’s not possible to delete more than 1000 rows at once.
  2. Deleting rows is very CPU intensive: to delete 1000 rows at once consumes two minutes of the 6.5 CPU hours a free app is allowed in a day.
  3. There is no programatic way to see how much of your CPU quota remains, so you can’t automate a deletion task without the risk of depleting your CPU.

I reasoned that, if deleting 1000 rows used two minutes of my daily 6.5 CPU hours, I could run my ‘TidyUpHandler’ every eight minutes and not take my app offine. After many hours of gradual purging, the stale data was cleared and Twitty City was brought within disk and CPU quota. The TidyUpHandler still runs every eight minutes, and as a result storage is now stable at around 12% of capacity.

If you aren’t constantly maintaining your data, GAE is quite unforgiving. A gigabyte of free storage is unparalleled in web app hosting, but if your app’s datastore grows quickly by itself you need to take measures to ensure it won’t get out of hand, and you need to do so before you’re near quota.

(Or, you could just pay for extra storage/CPU, but where’s the fun in that?)

How to add the Google +1 Button to your WordPress theme

Google Plus One

The Google +1 button hit the web earlier today, and I couldn’t resist the opportunity to put it on WordPress. Follow these steps and you can enable Google +1 on WordPress’s default TwentyTen theme:

  1. First we need to add the relevant JavaScript to the theme’s header file (by default at /wp-content/themes/twentyten/header.php). Just before the closing </head> tag, add the following:

  2. We then need to adjust loop-single.php so that individual posts will have the +1 button on them. Immediately after the <h1 class=’entry-title’> line (likely on line 26), add the following:
  3. Finally, add the following to loop.php immediately after the final <h1 class=’entry-title’> (likely on line 132), so that posts on the homepage will have +1 buttons:

And that’s it! Now you’re ready to make good on the Internet’s latest e-validation button.

How to test UTF-8 email subject lines in Symfony

Succcess Tests with Lime

One of our clients needs to send emails with UTF-8 subject lines. Swift Mailer, which Symfony uses, makes this easy to do, but writing functional tests for it is another story. The problem is that, if non-ASCII characters are present in the subject line, Swift Mailer encodes them as required by RFC 2047 for example, a string like “Transformación” turns into “=?utf-8?Q?Transformaci=C3=B3n?=”. How do we know that this is the right subject line?

There’s no immediately-available “convert this into a RFC 2047-compliant subject line” function, but we can get around this by making use of Swift_Message->toString and some choice preg functions.

First, create a dummy message containing the appropriate header. Swift_Message->toString() will contain our encoded header:

Next, we have to extract our encoded header I’ve used preg_match() and a fairly simple regex.

Finally, now that the encoded header has been extracted, we need only wrap it in preg_quote() (because checkHeader() uses regex syntax) and test away.

Voila - now we can be confident that our test script can deal with Unicode email subjects. If you can think of a way to make this simpler, or more robust, let me know in the comments.

JamBox - a music box for the web

Update: As of October 2011 JamBox has been ported to use Socket.IO and is hosted on Heroku, which does not presently support WebSockets. The information in this blog post is therefore (mostly) out of date - the God process is no longer around, and JamBox now works in most major browsers.

JamBox logo

My day job making content-managed websites is fun, and it pays the bills, but sometimes it’s nice to do something a little unconventional. So, I was thrilled when we had a “dev day” to write apps of our choosing in any language we wanted. Inspired by Jeff Kreeftmeijer’s node.js demo site (which shows every other visitor’s cursors in realtime) and Ron Winter’s Flash drum kit, I set about making a supercharged collaborative drumkit: JamBox.

JamBox’s premise is simple - each time you press a key, a sound plays, and everybody else on JamBox hears it. There are drum sounds, an extremely cheesy lead synth, a chunky slap bass, and various percussion noises (cowbell, bongos, DJ scratches, a dog woofing), so multiple users can play together and coordinate an actual jam session.

First, I wrote a single-user JavaScript drumkit - each sound is preloaded in the browser, and the keystroke listeners are bound. I then drew from Mr. Kreeftmeijer’s sample code to write the multi-user functionality. Crucially, I had plenty of time at the end of the hackday to spend in front of my favourite music-making tools, picking the right instruments and sampling them to perfection.

A downside is that JamBox relies on WebSockets, which aren’t supported by many browsers - not to mention, only a few browsers support .mp3 playback. As a result, JamBox has an “only works in Chrome/Safari” warning. Eventually I’d like to rewrite it to use the non WebSocket-specific socket.io library, and to switch between using .ogg/.mp3 files depending on the browser’s capabilities.

Another challenge was posed by the JamBox process’s tendency to stop running. Thanks to some sample code (again from Jeff Kreeftmeijer) I configured a God process to restart JamBox whenever it crashes.

All in all, I’m very pleased with the JamBox experience. I had a lot of fun making it, and even more fun making phat beats with it. Many thanks to White October for letting me build JamBox on company time, Jeff Kreeftmeijer for providing inspiration and generous sample code, and to my coworkers for helping me test and showing their musical talents.