Using a Python timeout decorator for uploading to S3
At work we are uploading many images to S3 using Python's boto library. However we are experiencing a RequestTimeTooSkewed error once every 100 uploads on average. We googled, but did not find a solution. Our system time was in sync and our file sizes were small (~50KB).
Since we couldn't find the root cause, we added a watchdog timer as a bandaid solution. We already use a retry decorator to retry uploads to S3 when we get a 500 Internal Server Error response. To this we added a timeout decorator which cancels the S3 upload if it takes ...
... read more »How to list attributes of an EC2 instance with Python and boto
Here's how to find out information about your Amazon EC2 instances using the Python boto library.
Install boto
- Install pip
- Install boto
sudo pip install boto
Example
from pprint import pprint
from boto import ec2
AWS_ACCESS_KEY_ID = 'XXXXXXXXXXXXXXXXXX'
AWS_SECRET_ACCESS_KEY = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
ec2conn = ec2.connection.EC2Connection(AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY)
reservations = ec2conn.get_all_instances()
instances = [i for r in reservations for i in r.instances]
for i in instances:
pprint(i.__dict__)
break # remove this to list all instances
Results:
{'_in_monitoring_element': False,
'ami_launch_index': u'0',
'architecture': u'x86_64',
'block_device_mapping': {},
'connection': EC2Connection:ec2.amazonaws.com,
'dns_name': u'ec2-xxx-xxx-xxx-xxx.compute-1.amazonaws.com',
'id': u'i-xxxxxxxx',
'image_id': u ...
... read more »On using Python, the Digg API, and simplejson
Here are some quick notes on using the
Digg API
with a Python script. Note, there is a
Python
toolkit for Digg but I just used urllib2
and the Digg API endpoints for the sake of simplicity.
I wanted the output in JSON format so I specified the response type as JSON. To decode JSON directly to a Python data structure, I used simplejson.
Here is a simple example which returns the JSON output for
the Digg story
Dell
vs. Apple: This Time it's Personal
which has a "clean title" of
Dell_vs_Apple_This_Time_it_s_Personal.
#!/usr/bin/env python
import urllib2
APPKEY ...How to use gnip-python to retrieve activity from Twitter, Delicious, Digg, etc.
- Create an account at http://www.gnipcentral.com/
- Download gnip-python from github.com.
- Unpack it:
$ tar -zxvf gnip-gnip-python-028364a70bd40dda0069ecdd3e7f6fff23bb985e.tar.gz - Move it to your example directory:
$ mkdir ~/src/python/gnip-example $ mv gnip-gnip-python-028364a70bd40dda0069ecdd3e7f6fff23bb985e/*.py ~/src/python/gnip-example
- Create an example file called
~/src/python/gnip-example/gnip-example.py:#!/usr/bin/env python from gnip import * gnip = Gnip("yourgniplogin@email.com", "yourpassword") for publisher in ["twitter", "digg", "delicious"]: activities = gnip.get_publisher_activities(publisher) print print publisher for activity in activities[:5]: print activity
- Run it:
$ python gnip-example.py
And get the following results:twitter [derricklo, 2008-08-01T22:49:59+00:00, tweet, http://twitter.com ...
About
I'm Eliot and this is my notepad for programming topics such as Python, Django, Ubuntu, Emacs, etc... more »
Search Blog
Tags
-
algorithms
(4)
-
aws
(8)
-
blogproject
(20)
-
c_cplusplus
(12)
-
cardstore
(8)
-
colinux
(2)
-
concurrency
(9)
-
conkeror
(2)
-
cygwin
(18)
-
datastructures
(15)
-
datetime
(3)
-
dell
(3)
-
django
(39)
-
emacs
(20)
-
files_directories
(10)
-
install_setup
(7)
-
javascript
(3)
-
keyboard
(6)
-
matplotlib
(5)
-
mercurial
(4)
-
nginx
(2)
-
preferences
(8)
-
processes
(3)
-
pyqt
(18)
-
python
(122)
-
ratpoison
(3)
-
regexes
(5)
-
rsync
(3)
-
softwaretools
(17)
-
sql
(13)
-
ssh
(7)
-
subversion
(6)
-
twisted
(6)
-
ubuntu
(60)
-
urxvt
(5)
-
vxworks
(25)
-
webservices
(4)
-
wmii
(7)
Blogroll
- Adam Gomaa
- Alex Clemesha
- Amir Salihefendic
- Armin Ronacher
- David Beazley
- David Ziegler
- Duncan McGreggor
- Gareth Rushgrave
- Glyph Lefkowitz
- Guido van Rossum
- Ian Bicking
- Jacob Kaplan-Moss
- James Bennett
- James Tauber
- Jesper Noehr
- Matt Harrison
- Nikolay Kolev
- Parand Darugar
- Peter Baumgartner
- Peter Bengtsson
- Rob Hudson
- Simon Willison
- Will McGugan