Context

It's been nearly two months since I first posted about getting started with Google App Engine. The post discussed web services and the ways in which parts of your application can be outsourced to larger, more scalable platforms - with a specific focus on Google App Engine (GAE). At that time I put up two web services on Github, AppImage and AppMail which leverage the GAE infrastructure to handle image upload/manipulation and mail delivery.

In the months since that post a few projects at work have progressed and I've found myself giving more attention to AppImage - exploring how it could be further improved. When I continued playing around with AppImage, it quickly became obvious that it was being severely crippled by the 1MB image resizing limit imposed within the GAE environment. Many of our users were having avatar images rejected for being too large - which is a confusing error for users because almost every photo out of digital camera is bigger than 1MB. Researching on the web I discovered that there are many other devs being impacted by this limit (StackOverflow & Google Code Issue 1422).

I set out searching for a solution and happened to stumble across a past post on the Google App Engine Blog which describes a new Blobstore API that supports file upload and storage of up to 50MB. An idea for a solution came together by combining that knowledge with the Python Images API Overview documentation which outlines that you can input an image larger than 1MB to the Images API only if it is retrieved via the Blobstore (compared to being held in memory or from the Datastore). These pieces of information were the building blocks and, after a few hurdles, a solution was formed and an update was made to AppImage.

To the point

In short, AppImage is now at Version 2.0. It supports large file resizing and now has XML responses to provide a more structured output from the service. The update included modifications to the AppImage GAE app and the AppImage PHP Library. A full release history and documentation is maintained in the README on Github.

How the solution works

The solution essentially works by combining two components of the GAE API - The Blobstore and the Image Service allowing large images from the Blobstore to be resized. The original solution used one path, however that path is now split depending on the size of an image. If the image is greater than 1MB then the process is two step - upload to blobstore, then process the upload using a blob_key as reference.

Roadblocks

I thought it would be good to talk about some of that things that had me hitting my head against the wall for a many hours during development, not just for interest but to hopefully save you from similar headaches.

Blobstore Handlers

The biggest roadblock I hit was dealing with the Blobstore Handlers and getting a response from them when using the PHP library. The problem lied in a clause I'd overlooked in the GAE documentation.

Finally, the handler must return a headers-only, redirect response (301, 302, or 303), typically a browser redirect to another URL handler.

It all came together having read that and then updating the UploadHandler in AppImage to redirect to a response page (that just dumps the associated blob key) and then updating appimage.rest.php to include curl_setopt($c, CURLOPT_FOLLOWLOCATION, true); in the _httpPost method.

Output Filesize

The API limitation of only allowing output data from the Image Serive to be < 1MB remains in GAE and was another roadblock during development. Now that the Blobstore API (which accepts files up to 50MB) has been integrated into AppImage you'd hope you were able to resize any image and then store it in the Datastore. However an exception - RequestTooLargeError: The request to API call images.Transform() was too large. - is raised whenever you try to transform an image that results in the output being over 1MB. The solution to this was to place a "max dimension" property to the Image Models (Avatar and Photo).

The "max dimension" property acts as an initial transformation that reduces any image passed into the specified dimension (for example purposes we'll use 600px). This however only holds up based on the assumption that any image when transformed to something like 600px will be easily less than 1MB - which should in most cases be true.

The initial transformation also means that you aren't storing anything more than you need, for instance if a user uploads a 10MB image (thousands of pixels in dimension), then this is immediately reduced to the specified "max dimension" (which should be at least equivalent to the largest dimension used in your web app), afterwards you delete the original 10MB image and you no longer have unneeded data chewing up your GAE Quotas.

Get The Code

From now on the source code for AppImage and AppMail will only be provided as Github downloads because I'm hoping they'll be regularly updated by myself and anyone else wanting to contribute.

Things to Note

These are just a few bits and pieces of notes that should help when using AppImage. If I've missed anything you think should be included please let me know in the comments.

Blobstore Is A Paid Service

You'll have to enable billing to get access to the Blobstore API. It's a surprisingly simple process. I set it up so that the maximum I can possibly get charged a day is $1 and that's only if I exceed the already quite generous quotas given.

Changed Function Declarations

The PHP library's main upload method has been updated to require a filesize argument to be passed. The old declaration was function upload($type, $file, $img_id) and the new one is function upload($type, $file, $filesize, $img_id). If you are updating to AppImage 2.0 then you'll need to update upload method calls accordingly.

XML Responses

AppImage now returns XML instead of dumping strings. The PHP library has been updated to read the new XML response correctly, however incase you're developing in another language I've included two examples to illustrate the response structure.

// successful upload returns the Datastore image object key
<response>
  <status>app.success</status>
  <message>ag1hcHBpbWFnZWRlbW8ycg0LEgZBdmF0YXIYkQgM</message>
</response>

// an invalid API key and/or IP address used to call the web service
<response>
  <status>app.forbidden</status>
  <message>You don't have permission to perform this action: ...</message>
</response>

Logging

You will notice the GAE AppImage code has been littered with logging method calls, this was a new feature for me in the GAE API. I found it particularly useful for debugging, however you will want to remove/comment out the logging code when deploying to production.

Multiple Thumbnails

When looking into performance and image quality for one of our projects, it seemed a good idea to generate thumbnails for each image to corresponded to the dimensions of images required by our web app. This will hopefully increase performance by users only downloading what they need to, although it will also depend on the caching you've deployed - a future upgrade to AppImage will include support for the GAE Memcache API. I've left the code in the Image Models that illustrates how multiple thumbnails can be implemented, but this can easily be added to or removed depending on your situation.

That's all folks

In the end this solution was quite simple. It was only because of a series of dependencies and hitting a few roadblocks that made me think it was something important to share with others. I'm hoping it's not only useful as a working web service, but also simply as a chunk of code that can be an example of how to get large image resizing working in any GAE applications you might already have. If you do find it useful or if you implement AppImage in your application, I'd be keen to hear about it.

As always, Enjoy.