Pycon 2012 talks that I saw that I enjoyed
The Pycon 2012 videos are up at pyvideo.org. Here are some of the talks I enjoyed that I saw. I know I probably missed some great talks so I will try to watch more online. Let me know if there are some that I should not miss.
Favorite talk of the conference
-
Saturday Keynote: Let's Talk About pypy
by David Beazley.
It was awesome! See also his
post Pycon blog post
Other great talks (in chronological order)
-
The Art of Subclassing by
Raymond Hettinger.
Great instruction on a topic I struggle with (object-oriented code).
- Stop Writing Classes by ...
A unique Python redis-based queue with delay
This is a simple Redis-based queue. Two features that I needed were uniqueness (i.e. if an item exists in the queue already, it won't be added again) and a delay, like beanstalkd, where an item must wait a specified time before it can be popped from the queue. There are a number of other Redis-based queues that have many more features but I didn't see one that had these two features together. This 50-line class works for my needs. It may or may not work for you. Feel free to copy this and build on it ...
... read more »Python gnupg (GPG) example
python-gnupg is a Python package for encrypting and decrypting strings or files using GNU Privacy Guard (GnuPG or GPG). GPG is an open source alternative to Pretty Good Privacy (PGP). A popular use of GPG and PGP is encrypting email. For more information, see the python-gnupg documentation. Another option for encrypting data from Python is keyczar.
Install¶
This installs the Ubuntu GPG package, creates a test user, and installs the Python package, python-gnupg. This was installed on Ubuntu 10.10 Maverick Meerkat.
$ sudo apt-get install gnupg
$ sudo adduser testgpguser
$ sudo su testgpguser
$ cd
$ virtualenv --no-site-packages venv
$ source venv/bin/activate ...Using curl over ftp took 3+ minutes for a 4 byte file w/ EPSV
It took over 3 minutes to download a 4 byte file with curl via ftp. It took less than a second with wget.
$ time curl -o testfile.txt -u myusername:mypassword ftp://ftp.myserver.com/path/to/testfile.txt
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4 0 4 0 0 0 0 --:--:-- 0:03:09 --:--:-- 0
real 3m9.411s
user 0m0.000s
sys 0m0.050s
Running curl with the verbose option¶
Running curl with the verbose option showed it was failing in "Extended Passive Mode".
$ curl -v -o testfile.txt ...Some more python recursion examples
I had a number of yaml files that contained passwords that needed encrypting. After parsing the yaml file with pyyaml, the data looked something like this:
EXAMPLE_DATA = {
'jobs': [{'frequency': '* * * * *',
'jobconfig': [{'config': [('*',
{'maxspeed': 1048576,
'password': 'onesecretpassword',
'port': 22,
'url': 'basset://basset1.domain.com/tootsiepop/123.csv',
'username': 'myusername'})],
'hasbro': 'basset'},
{'config': [('*',
{'field_delim': ',',
'field_line': True,
'no_blanks': True,
'quote_char': '"'})],
'hasbro': 'pen'},
{'config': [('*',
{'db_database': 'mydatabase',
'db_host': 'myhost',
'db_password': 'anothersecretpassword',
'db_table': 'mytable',
'db_user': 'myuser'})],
'hasbro': 'dart'}],
'jobdesc': 'Data from tootsiepop',
'jobname': 'tootsiepop',
'max_records_fail': '110%',
'min_failure_time': '1000y'}],
'vendor': 'tootsiepop'}
Print all leaf nodes¶
Here is a recursive function that prints all the leaf nodes of my ...
... read more »How to remove ^M characters from a file with Python
Use the following Python script to remove ^M (carriage
return) characters from your file and replace them with newline
characters only. To do this in Emacs,
see my notes here.
remove_ctrl_m_chars.py:
import os
import sys
import tempfile
def main():
filename = sys.argv[1]
with tempfile.NamedTemporaryFile(delete=False) as fh:
for line in open(filename):
line = line.rstrip()
fh.write(line + '\n')
os.rename(filename, filename + '.bak')
os.rename(fh.name, filename)
if __name__ == '__main__':
main()
Run it
$ python remove_ctrl_m_chars.py myfile.txt
Documentation
-
Built-in Functions —
open() - Built-in Types — File Objects
-
tempfile— Generate temporary files and directories - String Methods ...
Notes on debugging ssh connection problems
- Run the ssh client in verbose mode
$ ssh -vvv user@host - On the server, check auth.log for errors
$ sudo tail -f /var/log/auth.logOn Red Hat, it's
/var/log/secure - For more debugging info, (assuming you have control of the ssh server)
run the sshd server in debug mode on another port
Then specify the port,
$ sudo /usr/sbin/sshd -ddd -p 33333-p 33333with the ssh client. e.g.$ ssh -vvv -p 33333 user@host
Commands run on Ubuntu 10.04
sftp error: Received message too long 170160758¶
Problem was in the .bashrc. See ...
... read more »Example using git bisect to narrow in on a commit
I learned about
git bisect
from this Stack Overflow poll:
What are your favorite git features or tricks?
I thought it was so cool that I wanted to share an example.
After upgrading from Django 1.2.3 to Django 1.3, something broke on the website
I was working on. To figure out what was wrong, I used git bisect
to find the Django revision that introduced the relevant change. I cloned the
Django github repo and
pip install -e'd it into my virtualenv. Then
I used git bisect as follows:
- Start git bisect
$ git bisect start$ git ...
Remove leading and trailing whitespace from a csv file with Python
I'm reading a csv file with the Python csv module and could not find a setting to remove trailing whitespace. I found this setting, Dialect.skipinitialspace, but it I think it only applies to leading whitespace. Here's a one-liner to delete leading and trailing whitespace that worked for me.
import csv
reader = csv.DictReader(
open('myfile.csv'),
fieldnames=('myfield1', 'myfield1', 'myfield3'),
)
# skip the header row
next(reader)
# remove leading and trailing whitespace from all values
reader = (
dict((k, v.strip()) for k, v in row.items() if v) for row in reader)
# print results
for row in reader ...Example parsing XML with lxml.objectify
Example run with lxml 2.3, Python 2.6.6 on Ubuntu 10.10
from lxml import objectify, etree
xml = '''
<dataset>
<statusthing>success</statusthing>
<datathing gabble="sent">joe@email.com</datathing>
<datathing gabble="not sent"></datathing>
</dataset>
'''
root = objectify.fromstring(xml)
print root.tag
print root.text
print root.attrib
# dataset
# None
# {}
print root.statusthing.tag
print root.statusthing.text
print root.statusthing.attrib
# statusthing
# success
# {}
for e in root.datathing:
print e.tag
print e.text
print e.attrib
print e.attrib['gabble']
# datathing
# joe@email.com
# {'gabble': 'sent'}
# sent
# datathing
# None
# {'gabble': 'not sent'}
# not sent
for e ...About
I'm Eliot and this is my notepad for programming topics such as Python, Django, Ubuntu, Emacs, etc... more »
Search Blog
Tags
-
![[algorithms feed]](http://static.saltycrane.com/image/icon_feed_gray_5.png)
algorithms
(6)
-
android
(2)
-
aws
(10)
-
blogproject
(20)
-
c_cplusplus
(12)
-
cardstore
(8)
-
colinux
(2)
-
concurrency
(13)
-
conkeror
(2)
-
core
(2)
-
cygwin
(17)
-
datastructures
(15)
-
datetime
(4)
-
decorators
(4)
-
django
(41)
-
emacs
(22)
-
files_directories
(12)
-
git
(6)
-
hardware
(6)
-
install_setup
(8)
-
javascript
(3)
-
keyboard
(9)
-
matplotlib
(6)
-
mercurial
(4)
-
nginx
(2)
-
persistence
(6)
-
preferences
(7)
-
processes
(4)
-
pyqt
(18)
-
python
(157)
-
ratpoison
(3)
-
regexes
(6)
-
rsync
(3)
-
softwaretools
(17)
-
sql
(14)
-
ssh
(12)
-
subversion
(6)
-
twisted
(7)
-
ubuntu
(66)
-
urxvt
(5)
-
vxworks
(25)
-
webdev
(8)
-
wmii
(7)
Blogroll
- Adam Gomaa
- Alex Clemesha
- Amir Salihefendic
- Armin Ronacher
- David Beazley
- David Ziegler
- Duncan McGreggor
- Gareth Rushgrave
- Glyph Lefkowitz
- Guido van Rossum
- Ian Bicking
- Jacob Kaplan-Moss
- James Bennett
- James Tauber
- Jesper Noehr
- Marty Alchin
- Matt Harrison
- Nikolay Kolev
- Parand Darugar
- Peter Baumgartner
- Peter Bengtsson
- Rob Hudson
- Simon Willison
- Will McGugan