Context

Recently at work we've been researching ways to improve the performance and flexibility of our web apps. We've begun trending towards outsourcing CPU intensive tasks and storage to bigger players such as Amazon and Google, basically to ensure that our web server is doing exactly (and only) what its supposed to do - serve web pages. Keeping that in mind, some tasks that large apps often need to perform but don't fall in the category of page serving are:

  • Email (sending activation links, lost passwords, updates)
  • Image Manipulation (avatar, photo uploads)
  • Content Serving (css, images, javascript, other media types)
  • Database (CRUD operations, etc)

All these tasks are of course critical for an application, however you can achieve performance gains by wrapping these components up into individual web services that can each scale up and down based on need. By doing so you also get flexibility because these services (if well designed) can now "plug and play" into other applications. The services act as their own applications exposing an API for the main application to interact with. Of course these ideas aren't new, yet its an interesting mindset that I've been exploring when developing web apps.

There are some potential problems to this approach though. Splitting the main app into multiple systems means there is, not surprisingly, now multiple systems to manage. This can be a problem if they are written in different languages or use different platforms (ie, one using Java on Amazon EC2, the other using Python on Google App Engine). There can also be overhead in developing a completely encapsulated system (ie, to handle email), compared to using an inbuilt quick bit of code to manage it in the main app which could reuse existing exception handling, framework methods, or configuration information.

All aspects should be considered, the approach we've been taking for our latest web project at work is to build these encapsulated services to hopefully make scaling easier in the future. I've written up some information about two that I've been working on and in the spirit of sharing code I'm hoping to seek thoughts and ideas for improvements from yourself and others. Email and Image Manipulation were two areas we wanted to outsource and Google App Engine has proven to be a solid, suitable platform.

To the point

The two services developed are Google App Engine (GAE) Python scripts that I've written/modified to handle Email (AppMail) and Image Manipulation (AppImage). The original source for AppImage was Appspotimage which is an excellent example of a GAE app that made it easy to "learn from code" the groundings needed for Python and GAE development.

AppMail

Almost all web apps need to send email, however its important that the process of sending emails doesn't hold up a user. Also what do you do if an error occurs? Will an exception be raised that alerts the user? Not a good idea for emails, its not the user's problem, its ours and we should manage emails in a way that is invisible to the user.

AppMail is a very small app that leverages the inbuilt mail service and queuing that the GAE platform provides. One of the advantages of this mail service is that its asynchronous - important when we're aiming to be invisible to our users.

Sending is asynchronous: The mail.send_mail() function and the EmailMessage send() method transmit the message data to the mail service, then return. The mail service queues the message, then attempts to send it, possibly retrying if the destination mail server is unavailable. Errors and bounce messages are sent to the sender address for the email message.

Other advantages include:

  • Queuing: Our web app can behave the way it needs to, knowing that GAE will take care of queuing messages until there are available resources to send them.
  • Efficient code: No doubt the sending and queuing code is efficient having been written by the brains at Google and in Python.

AppImage

AppImage is a lightweight version of an open source script - Appspotimage. Appspotimage is a solid solution and has a bunch of manipulation tasks ready to go. However for our project we needed some customisations and data security, so discovering it was open source was a very welcome surprise.

GAE is a particularly powerful and feature rich environment for image manipulation. It has a full image service that is simple to utilise. Initially it was limited by 1mb limit per datastore record but only last week they increased that to 50mb which is important because AppImage keeps the original, transformed and thumbnail images in the one record and that was crippled under the previous limit.

AppImage includes two manipulation methods - resize and crop (more can be added by referencing Appspotimage code). It also uses two models "Avatar" and "Photo" to illustrate how the single script can be used to manage all the image assets in your web app.

Get the code

Below is a bundle containing the two apps and corresponding PHP libraries for interacting with them. If you're keen please feel free to contribute libraries for your programming language of choice.

AppMail & AppImage

GIT ZIP TAR

Getting started

  • Create an account at Google App Engine
  • Create new application(s) for the code you've downloaded above
  • Download GAE SDK for Python
  • In the SDK create a "New Application" to match each of the application names you added in the GAE web dashboard.
  • Replace the files that it auto-generates with the ones you've downloaded above
  • Modify app.yaml and the "AUTH" constant in main.py to use your IP Address and API Key
  • Deploy it
  • If you're using PHP: modify the config file and run the tests provided to check it all works.

About the inbuilt authentication

Both apps include an authentication aspect to stop others using your app, data, and bandwidth. The authentication works by requiring an API Key to be passed when making requests to certain application methods. This API Key is then looked up in the "AUTH" dictionary in main.py which has a corresponding IP Address. If the requesters IP Address matches the one corresponding the API Key provided then all is good and access is provided, if they don't match an exception is raised. The API Key can be made by either mashing your keyboard vigorously or better yet by visiting GRC Password Generator.

Something extra for fellow CakePHP devs

I've included below a few extra CakePHP files (includes a component, model, db migration and app_controller.php method) for implementing AppMail. The idea is to behave more like the internal "Email" component in the CakePHP framework - allowing you to use email templates in your "views/elements" folder.

The model and db migration have been included if you want to keep track of emails in your own database and use a daemon shell script to send them off to GAE. Using the database could give a speed increase if the daemon sent emails in bulk to GAE which had a corresponding "debulker" to iterate through and send the emails. This hasn't been written yet, but I'll continue researching whether there is any potential performance to be gained. If you have some thoughts on the matter, I'd be keen to hear them.

AppMail CakePHP

ZIP TAR

Article resources

That's all folks

There's a bit of code there to consume, however if you are in the fortunate situation of building your own web apps and you want to leverage third party infrastructure, hopefully this has provided insight in beginning to use Google App Engine.

Explore the code and checkout the resources - I've found Python and the GAE platform to be a fun way to get small yet powerful apps up in very little time. As always please jump on the comments if you have questions or just want to discuss ideas. Enjoy.