Sebastian Motraghi on life, tech, et cetera

The Limits of "Free" on Google App Engine

Update: Just my luck - I write 99% of a blog post about how I got around the limitations of Google App Engine’s free tier, and then I get an email announcing how Google will drastically change their price structure not in my favour. Looks like I’m not done trying to outsmart Google’s billing department…

Google App Engine’s free tier is great for small apps, but as I discovered when Twitty City reached a gigabyte of storage, its limitations can be extremely frustrating.

Twitty City ran well for its first few months, but after 400,000 tweets its datastore was full and needed to be cleaned up. Unlike the typical MySQL/PHP setup, GAE does not give the user the ability to delete from the datastore with SQL commands: data must be obtained programmatically. The only way to delete several rows is to select them in your program’s code, iterate over them, and delete them individually. This would be tolerable, except for three factors:

  1. It’s not possible to delete more than 1000 rows at once.
  2. Deleting rows is very CPU intensive: to delete 1000 rows at once consumes two minutes of the 6.5 CPU hours a free app is allowed in a day.
  3. There is no programatic way to see how much of your CPU quota remains, so you can’t automate a deletion task without the risk of depleting your CPU.

I reasoned that, if deleting 1000 rows used two minutes of my daily 6.5 CPU hours, I could run my ‘TidyUpHandler’ every eight minutes and not take my app offine. After many hours of gradual purging, the stale data was cleared and Twitty City was brought within disk and CPU quota. The TidyUpHandler still runs every eight minutes, and as a result storage is now stable at around 12% of capacity.

If you aren’t constantly maintaining your data, GAE is quite unforgiving. A gigabyte of free storage is unparalleled in web app hosting, but if your app’s datastore grows quickly by itself you need to take measures to ensure it won’t get out of hand, and you need to do so before you’re near quota.

(Or, you could just pay for extra storage/CPU, but where’s the fun in that?)