Card store project #4: Notes on using Amazon's CloudFront

Date: 2008-12-27 | Modified: 2009-10-21 | Tags: aws, cardstore, django | 5 Comments

I haven't been keeping up with the current events very well recently, but I haven't noticed a lot of people using Amazon's S3 or CloudFront with Django on VPS hosting. Though there is Adrian's post from 2006, I see more articles about serving media files with lighttpd or, more recently, nginx. Is a CDN unnecessary for our needs? I thought it'd be good to take some load off my VPS server since I need all the memory I can get for my Django web server and database. But maybe web servers such as nginx are so lightweight it doesn't make much of an impact? I didn't think the cost would be too much-- on this blog, I'm only paying about $0.10/month for S3 services to serve my static media. Of course, there isn't a lot of static media to serve on this blog, but it still seems like it would be a fraction of the $20/month I'm paying for VPS at Slicehost. It may be the convenience factor-- because every time I update a static file, I then have to upload it to S3. This is even more inconvenient for files uploaded through the admin interface. I think some people have probably solved this already... maybe using Django signals. Maybe it is a combination of all these things. Please let me know what you think. If you're not using S3/CloudFront, why aren't you?

Well I went ahead and gave CloudFront a try since it is so easy. My card store project website seems to be somewhat faster than before. Please check it out here. I'm still not sure if I should be happy with the site's speed though. I did a quick memcached install, but I don't think I've configured it properly. I will probably need to revisit that. Anyways, here are my notes on using CloudFront with my Satchmo store.

Sign up for S3

Sign up for Amazon Web Services
Sign up for Simple Storage Service
Take note of your "Access Key ID" and your "Secret Access Key" under "Your Account", "Access Identifiers"

Get S3 Python library

Download the Amazon S3 Python library
Unpack it, and put s3-example-libraries/python/S3.py somewhere on your Python path.

Create a S3 bucket

Create a file named create_bucket.py:

import S3

ACCESS_KEY = 'myaccesskey'
SECRET_KEY = 'mysecretaccesskey'
BUCKET_NAME = 'handsoncards'

conn = S3.AWSAuthConnection(ACCESS_KEY, SECRET_KEY)
conn.create_bucket(BUCKET_NAME)

Run it:
```
python create_bucket.py
```

Upload files to S3

Download Adrian's S3 upload script and save it to /srv/HandsOnCards/handsoncards/bin/update_s3.py
Edit the script with the correct values for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and BUCKET_NAME.

Upload files. (Assumes static directory is linked to /var/www/site_media).

cd /var/www
find -L site_media | grep -v '~$' | python /srv/HandsOnCards/handsoncards/bin/update_s3.py
find -L admin_media | grep -v '~$' | python /srv/HandsOnCards/handsoncards/bin/update_s3.py

Set up CloudFront

Sign up for CloudFront
Get the S3 Fox firefox plugin
Click "Manage Accounts" and enter access key and secret key
Right click on your bucket (handsoncards) and select "Manage Distributions" Enter a "Comment" and optional CNAME, then click "Create Distribution".
Wait a while while the distribution is created. Take note of the "Domain Name". For me it is: http://d16z1yuk7jeryy.cloudfront.net
Click the refresh button until the "Status" says "Deployed"

Update settings and templates to use CloudFront

In settings.py set MEDIA_URL and ADMIN_MEDIA_PREFIX as follows:

MEDIA_URL = 'http://d16z1yuk7jeryy.cloudfront.net/site_media/'
ADMIN_MEDIA_PREFIX = 'http://d16z1yuk7jeryy.cloudfront.net/admin_media/'

In your base.html template and all other templates, replace /site_media/ with http://d16z1yuk7jeryy.cloudfront.net/site_media/.

Update 2009-06-08: Add "Expires" headers

For better performance, it is good to add a far-future "Expires" header to static content on S3. To do this I modified Adrian's script to set the "Expires" header to be one year in the future as shown below. Thanks to orip for this tip.

from datetime import datetime, timedelta
import mimetypes
import os.path
import sys
import S3 # Get this from Amazon

AWS_ACCESS_KEY_ID = 'CHANGEME'
AWS_SECRET_ACCESS_KEY = 'CHANGEME'
BUCKET_NAME = 'CHANGEME'

def update_s3():
    conn = S3.AWSAuthConnection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
    for line in sys.stdin:
        filename = os.path.normpath(line[:-1])
        if filename == '.' or not os.path.isfile(filename):
            continue # Skip this, because it's not a file.
        print "Uploading %s" % filename
        filedata = open(filename, 'rb').read()
        expires = datetime.utcnow() + timedelta(days=365)
        expires = expires.strftime("%a, %d %b %Y %H:%M:%S GMT")
        content_type = mimetypes.guess_type(filename)[0]
        if not content_type:
            content_type = 'text/plain'
        conn.put(BUCKET_NAME, filename, S3.S3Object(filedata),
            {'x-amz-acl': 'public-read', 
             'Content-Type': content_type,
             'Expires': expires,
             })

if __name__ == "__main__":
    update_s3()

For more information:

Update 2009-10-21: Add CNAME record

Go to your DNS Zone manager and add a CNAME record with the following parameters:

Type: CNAME
Name: static
Data: d16z1yuk7jeryy.cloudfront.net. (Don't forget the period at the end!)
TTL: whatever you want. I left it at 86400

Now wherever I previously would have used http://d16z1yuk7jeryy.cloudfront.net, I can replace it with http://static.handsoncards.com.

5 Comments

#1 Andy commented on 2009-01-08:

Thanks for the post! If you are on Windows try CloudBerry Explorer for Amazon S3. It makes managing files in S3 and CloudFront EASY http://cloudberrylab.com/

#2 Tom Gruner commented on 2009-04-24:

Thanks for the nice list of instructions! I am currently setting up a vps with Django and cloud front, but somehow I missed you had to have the s3 part.

For me the only advantage is not that it takes the load of the vps, but also that the target audience of the website will be in China, and the cloud front will make the media arrive quickly even though my vps is in the US. And the price is quite reasonable from my pov considering the advantage it offers.

#3 Eliot commented on 2009-04-24:

Hi Tom, thanks for your comment. You make a good point. Since my target audience for this project is the US, it doesn't benefit me as much. I'm actually trying to set up nginx for my static files right now. This seems to be the popular choice for my more common use case.

#4 orip commented on 2009-06-08:

Thanks for the instructions!

A question about caching - I noticed that on http://handsoncards.com/ the images on cloudfront aren't cached at all.

Couldn't you add a far-future expires header by modifying update_s3.py and setting the "Expires" or "Cache-Control" HTTP headers?

#5 Eliot commented on 2009-06-08:

orip,
Thanks for the tip! I modified the script to set the "Expires" header to be one year in the future. See my update above.


Name:	Required
Email:	Required, but not displayed
Website:	Optional
Comment:	Format using Markdown. (No HTML.) Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after) Code span: surround with backticks Blockquotes: prefix lines to be quoted with > Links: <URL> Links w/ description: [description](URL)

SaltyCrane Blog