Posts tagged "webservices" « SaltyCrane Blog

Using a Python timeout decorator for uploading to S3

Date: 2010-04-27 | Tags: aws, python, webservices | 3 Comments

At work we are uploading many images to S3 using Python's boto library. However we are experiencing a RequestTimeTooSkewed error once every 100 uploads on average. We googled, but did not find a solution. Our system time was in sync and our file sizes were small (~50KB).

Since we couldn't find the root cause, we added a watchdog timer as a bandaid solution. We already use a retry decorator to retry uploads to S3 when we get a 500 Internal Server Error response. To this we added a timeout decorator which cancels the S3 upload if it takes ...

How to list attributes of an EC2 instance with Python and boto

Date: 2010-03-08 | Modified: 2010-04-13 | Tags: aws, python, webservices | 5 Comments

Here's how to find out information about your Amazon EC2 instances using the Python boto library.

Install boto

Install pip
Install boto
```
sudo pip install boto
```

Example

from pprint import pprint
from boto import ec2

AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'

ec2conn = ec2.connection.EC2Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2conn.get_all_instances()
instances = [i for r in reservations for i in r.instances]
for i in instances:
    pprint(i.__dict__)
    break # remove this to list all instances

Results:

{'_in_monitoring_element': False,
 'ami_launch_index': u'0',
 'architecture': u'x86_64',
 'block_device_mapping': {},
 'connection': EC2Connection:ec2.amazonaws.com,
 'dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
 'id': u'i-xxxxxxxx',
 'image_id': u ...

On using Python, the Digg API, and simplejson

Date: 2008-08-19 | Modified: 2009-03-25 | Tags: python, webservices

Here are some quick notes on using the Digg API with a Python script. Note, there is a Python toolkit for Digg but I just used urllib2 and the Digg API endpoints for the sake of simplicity.

I wanted the output in JSON format so I specified the response type as JSON. To decode JSON directly to a Python data structure, I used simplejson.

Here is a simple example which returns the JSON output for the Digg story Dell vs. Apple: This Time it's Personal which has a "clean title" of Dell_vs_Apple_This_Time_it_s_Personal.

#!/usr/bin/env python

import urllib2

APPKEY ...

How to use gnip-python to retrieve activity from Twitter, Delicious, Digg, etc.

Date: 2008-08-01 | Modified: 2009-03-25 | Tags: python, webservices

Create an account at http://www.gnipcentral.com/
Download gnip-python from github.com.

Unpack it:

$ tar -zxvf gnip-gnip-python-028364a70bd40dda0069ecdd3e7f6fff23bb985e.tar.gz

Move it to your example directory:

$ mkdir ~/src/python/gnip-example
$ mv gnip-gnip-python-028364a70bd40dda0069ecdd3e7f6fff23bb985e/*.py ~/src/python/gnip-example

Create an example file called ~/src/python/gnip-example/gnip-example.py:

#!/usr/bin/env python

from gnip import *

gnip = Gnip("yourgniplogin@email.com", "yourpassword")

for publisher in ["twitter", "digg", "delicious"]:
    activities = gnip.get_publisher_activities(publisher)
    print
    print publisher
    for activity in activities[:5]:
        print activity

Run it:

$ python gnip-example.py

And get the following results:

twitter
[derricklo, 2008-08-01T22:49:59+00:00, tweet, http://twitter.com ...

SaltyCrane Blog