SaltyCrane Blog — Notes on JavaScript and web development

How to use Python's enumerate and zip to iterate over two lists and their indices.

enumerate- Iterate over indices and items of a list

The Python Cookbook (Recipe 4.4) describes how to iterate over items and indices in a list using enumerate. For example:

alist = ['a1', 'a2', 'a3']

for i, a in enumerate(alist):
    print i, a

Results:

0 a1
1 a2
2 a3

zip- Iterate over two lists in parallel

I previously wrote about using zip to iterate over two lists in parallel. Example:

alist = ['a1', 'a2', 'a3']
blist = ['b1', 'b2', 'b3']

for a, b in zip(alist, blist):
    print a, b

Results:

a1 b1
a2 b2
a3 b3

enumerate with zip

Here is how to iterate over two lists and their indices using enumerate together with zip:

alist = ['a1', 'a2', 'a3']
blist = ['b1', 'b2', 'b3']

for i, (a, b) in enumerate(zip(alist, blist)):
    print i, a, b

Results:

0 a1 b1
1 a2 b2
2 a3 b3

Recommended books

I love having a subscription to Safari Books Online. Currently my company provides a free subscription, but if I get a new job, I might consider subscribing myself. Since I get to browse a number of books at no cost, I thought I'd note which books are my favorites. (Note, I am not being paid by Safari Books Online.)

General Software

  • Structure and Interpretation of Computer Programs, Second Edition, Harold Abelson and Gerald Jay Sussman, MIT Press, ?year?
    I learned about this book through a job posting. It might bring you to tears if you get it. I'm only in the second chapter. It is used in an introductory Computer Science course at MIT. It uses Scheme (Lisp) to demonstrate concepts.
    Available free online at: http://mitpress.mit.edu/sicp/full-text/book/book-Z-H-4.html
    A free video lecture series is also available.

C

  • The C Programming Language, Second Edition, Brian W. Kernighan and Dennis M. Ritchie, Prentice Hall, 1988
    The definitive C book.

Python

  • Core Python Programming, Second Edition, Wesley J. Chun, Prentice Hall, September 18, 2006
    Usually I like O'Reilly books best, but I slightly prefer Chun's text to Learning Python.
    Available at Safari Books Online
  • The Django Book, Apress, December 2007
    I think this is the first official Django book.

SQLite

  • The Definitive Guide to SQLite, Mike Owens, Apress, May 2006
    I browsed a few SQL books but liked this one better than most. It has a good theory section.
    Available at Apress.com

Linux or related

  • X Power Tools, Chris Tyler, O'Reilly, December 15, 2007
    Lots of good information on the X Window System and more; easy to understand. I wish the basic Ubuntu or Linux books had some of this information.
    Available at Safari Books Online
  • SSH, The Secure Shell: The Definitive Guide, Second Edition, Daniel J. Barrett; Richard E. Silverman; Robert G. Byrnes, O'Reilly, May 10, 2005
    Available at Safari Books Online

Ruby

Javascript

  • Javascript: The Good Parts, Douglas Crockford, O'Reilly, May 2008
    This is the first Javascript book I read as I switched to Javascript and Frontend development. I was debating whether to read it since it was so old but I found it to be good and recommend it.
  • You Don't Know JS: ES6 & Beyond, Kyle Simpson, O'Reilly, December 2015
    This is the primary way I learned ES6. It is very detail oriented.
    Available free online on github
  • JavaScript Allongé, the "Six" Edition, Reg "raganwald" Braithwaite, Leanpub, 2016
    This was the third Javascript book I read and it was the most fun. It teaches functional programming concepts such as closures, shadowing, higher order functions, combinators, decorators, etc using ES6.
    Available free online at: https://leanpub.com/javascriptallongesix/read

Non technical

  • The Hitchhiker's Guide to the Galaxy by Douglas Adams
  • Calvin and Hobbes by Bill Watterson
  • Crime and Punishment by Fyodor Dostoyevsky
  • Mere Christianity by C. S. Lewis
  • Screwtape Letters by C. S. Lewis

See also

How to install Mercurial 1.0 on Cygwin

Mercurial 1.0 has been released and I just posted, How to install Mercurial 1.0 on Ubuntu Gutsy (or Hardy). Here are instructions for installing Mercurial 1.0 on Cygwin.

Install Mercurial 1.0
  1. Install Python Easy Install
    $ cd /tmp
    $ wget http://peak.telecommunity.com/dist/ez_setup.py
    $ python ez_setup.py
  2. Install Mercurial 1.0
    $ easy_install -U mercurial
  3. Trying hg version, gives me a ImportError: Permission denied error. Apparently, there is a Eggs/Cygwin problem. To fix it, make the dll files executable:
    $ chmod +x /usr/lib/python2.5/site-packages/mercurial-1.0-py2.5-cygwin-1.5.25-i686.egg/mercurial/*.dll
Try hg version again:
Mercurial Distributed SCM (version 1.0)

Copyright (C) 2005-2008 Matt Mackall  and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Install hgk extension
I'm not sure how to get hg view working in Cygwin. If anyone knows, please let me know.

How to install Mercurial 1.0 on Ubuntu Hardy

Mercurial 1.0 is out! Mercurial is a next generation "fast, lightweight source control management system" boasting conversions by notable projects NetBeans, OpenJDK, Globulation2, Xine, Mozilla, grml, and OpenSolaris.

From the release notes, version 1.0 has improvements for copying, renaming, removing, and merging. I want to use the most robust version possible, so I decided to upgrade. Ubuntu Gutsy's Mercurial version is 0.9.4, and even Hardy will only have 0.9.5. Luckily, Mercurial 1.0 is easy-installable. That makes it easy to install:


Install Mercurial 1.0
  1. If you don't already have Mercurial 0.9.4 installed, I'd suggest installing it because it will setup /etc/mercurial for using Mercurial extenstions.
    sudo apt-get install mercurial
  2. Install Python Easy Install, build tools, and Python header files.
    sudo apt-get install python-setuptools python-dev build-essential
  3. Install Mercurial 1.0
    sudo easy_install -U mercurial
That was easy! Test it out by typing hg version.
Mercurial Distributed SCM (version 1.0)

Copyright (C) 2005-2008 Matt Mackall  and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Upgrade to 1.0.1
Update 6/2/2008: Mercurial 1.0.1 has been released as a bugfix version. You can upgrade from 1.0 to 1.0.1 using Easy Install:
sudo easy_install -U mercurial

Install hgk extension
If you try the hg view command, you will get a sh: hgk: not found error. To fix this, I copied the hgk executable from the contrib directory in the source package to a directory in my path:
  1. Download and unpack the Mercurial source package.
    cd /tmp
    wget http://www.selenic.com/mercurial/release/mercurial-1.0.tar.gz
    tar zxvf mercurial-1.0.tar.gz
  2. Copy hgk somewhere on your path. E.g.:
    cp mercurial-1.0/contrib/hgk ~/bin
Note, if you get a /usr/bin/env: wish: No such file or directory error, it means you need to install the Tk package:
sudo apt-get install tk8.5

Other Notes
  • hbisect warning
    Per the Upgrade Notes, bisect is now a built-in command and the hbisect extension should not be used. If you get a extension 'hgext/hbisect' overrides commands: bisect warning, remove the hbisect= line from your configuration file (located at /etc/mercurial/hgrc.d/hgext.rc or ~/.hgrc).

  • Where are the files located?
    The hgext and mercurial files should be located at /usr/lib/python2.5/site-packages/mercurial-1.0-py2.5-linux-*.egg

  • stdio.h error during easy_install
    If you got the following error while trying to run easy_install, it probably means you don't have the "build-essential" package installed. $ sudo apt-get install build-essential.
    In file included from /usr/lib/gcc/i486-linux-gnu/4.2.3/include/syslimits.h:7,
                     from /usr/lib/gcc/i486-linux-gnu/4.2.3/include/limits.h:11,
                     from /usr/include/python2.5/Python.h:18,
                     from mercurial/mpatch.c:23:
    /usr/lib/gcc/i486-linux-gnu/4.2.3/include/limits.h:122:61: error: limits.h: No such file or directory
    In file included from mercurial/mpatch.c:23:
    /usr/include/python2.5/Python.h:32:19: error: stdio.h: No such file or directory
    /usr/include/python2.5/Python.h:34:5: error: #error "Python.h requires that stdio.h define NULL."
    /usr/include/python2.5/Python.h:37:20: error: string.h: No such file or directory
    /usr/include/python2.5/Python.h:39:19: error: errno.h: No such file or directory
    /usr/include/python2.5/Python.h:41:20: error: stdlib.h: No such file or directory
    /usr/include/python2.5/Python.h:43:20: error: unistd.h: No such file or directory
    /usr/include/python2.5/Python.h:55:20: error: assert.h: No such file or directory
    In file included from /usr/include/python2.5/Python.h:57,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pyport.h:7:20: error: stdint.h: No such file or directory
    In file included from /usr/include/python2.5/Python.h:57,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pyport.h:89: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Py_uintptr_t’
    /usr/include/python2.5/pyport.h:90: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Py_intptr_t’
    /usr/include/python2.5/pyport.h:113: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Py_ssize_t’
    /usr/include/python2.5/pyport.h:231:76: error: math.h: No such file or directory
    /usr/include/python2.5/pyport.h:238:22: error: sys/time.h: No such file or directory
    /usr/include/python2.5/pyport.h:239:18: error: time.h: No such file or directory
    /usr/include/python2.5/pyport.h:257:24: error: sys/select.h: No such file or directory
    /usr/include/python2.5/pyport.h:296:22: error: sys/stat.h: No such file or directory
    In file included from /usr/include/python2.5/Python.h:76,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pymem.h:50: warning: parameter names (without types) in function declaration
    /usr/include/python2.5/pymem.h:51: error: expected declaration specifiers or ‘...’ before ‘size_t’
    In file included from /usr/include/python2.5/Python.h:78,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/object.h:104: error: expected specifier-qualifier-list before ‘Py_ssize_t’
    /usr/include/python2.5/object.h:108: error: expected specifier-qualifier-list before ‘Py_ssize_t’
    /usr/include/python2.5/object.h:131: error: expected declaration specifiers or ‘...’ before ‘*’ token
    /usr/include/python2.5/object.h:131: warning: type defaults to ‘int’ in declaration of ‘Py_ssize_t’
    /usr/include/python2.5/object.h:131: error: ‘Py_ssize_t’ declared as function returning a function
    /usr/include/python2.5/object.h:131: warning: function declaration isn’t a prototype
    /usr/include/python2.5/object.h:149: error: ‘readbufferproc’ declared as function returning a function
    /usr/include/python2.5/object.h:150: error: ‘writebufferproc’ declared as function returning a function
    /usr/include/python2.5/object.h:151: error: ‘segcountproc’ declared as function returning a function
    /usr/include/python2.5/object.h:152: error: ‘charbufferproc’ declared as function returning a function
    /usr/include/python2.5/object.h:215: error: expected specifier-qualifier-list before ‘lenfunc’
    /usr/include/python2.5/object.h:229: error: expected specifier-qualifier-list before ‘lenfunc’
    /usr/include/python2.5/object.h:244: error: expected declaration specifiers or ‘...’ before ‘FILE’
    /usr/include/python2.5/object.h:258: warning: ‘struct _typeobject’ declared inside parameter list
    /usr/include/python2.5/object.h:258: warning: its scope is only this definition or declaration, which is probably not what you want
    /usr/include/python2.5/object.h:259: warning: ‘struct _typeobject’ declared inside parameter list
    /usr/include/python2.5/object.h:262: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/object.h:262: error: field ‘ob_size’ declared as a function
    /usr/include/python2.5/object.h:264: error: field ‘tp_basicsize’ declared as a function
    /usr/include/python2.5/object.h:264: error: field ‘tp_itemsize’ declared as a function
    /usr/include/python2.5/object.h:309: error: field ‘tp_weaklistoffset’ declared as a function
    /usr/include/python2.5/object.h:324: error: field ‘tp_dictoffset’ declared as a function
    /usr/include/python2.5/object.h:389: error: expected declaration specifiers or ‘...’ before ‘FILE’
    In file included from /usr/include/python2.5/Python.h:79,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/objimpl.h:97: warning: parameter names (without types) in function declaration
    /usr/include/python2.5/objimpl.h:98: error: expected declaration specifiers or ‘...’ before ‘size_t’
    /usr/include/python2.5/objimpl.h:228: error: ‘PyGC_Collect’ declared as function returning a function
    /usr/include/python2.5/objimpl.h:249: error: field ‘gc_refs’ declared as a function
    /usr/include/python2.5/objimpl.h:288: warning: parameter names (without types) in function declaration
    In file included from /usr/include/python2.5/Python.h:83,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/unicodeobject.h:55:19: error: ctype.h: No such file or directory
    /usr/include/python2.5/unicodeobject.h:118:21: error: wchar.h: No such file or directory
    In file included from /usr/include/python2.5/Python.h:83,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/unicodeobject.h:384: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/unicodeobject.h:385: error: field ‘length’ declared as a function
    /usr/include/python2.5/unicodeobject.h:447: error: ‘PyUnicodeUCS4_GetSize’ declared as function returning a function
    /usr/include/python2.5/unicodeobject.h:521: warning: type defaults to ‘int’ in declaration of ‘wchar_t’
    /usr/include/python2.5/unicodeobject.h:521: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ token
    /usr/include/python2.5/unicodeobject.h:539: warning: type defaults to ‘int’ in declaration of ‘wchar_t’
    /usr/include/python2.5/unicodeobject.h:539: error: expected ‘;’, ‘,’ or ‘)’ before ‘*’ token
    /usr/include/python2.5/unicodeobject.h:1102: error: ‘PyUnicodeUCS4_Tailmatch’ declared as function returning a function
    /usr/include/python2.5/unicodeobject.h:1114: error: ‘PyUnicodeUCS4_Find’ declared as function returning a function
    /usr/include/python2.5/unicodeobject.h:1123: error: ‘PyUnicodeUCS4_Count’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:84,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/intobject.h:24: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/intobject.h:38: warning: parameter names (without types) in function declaration
    /usr/include/python2.5/intobject.h:41: error: ‘PyInt_AsSsize_t’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:86,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/longobject.h:25: error: ‘_PyLong_AsSsize_t’ declared as function returning a function
    /usr/include/python2.5/longobject.h:26: warning: parameter names (without types) in function declaration
    /usr/include/python2.5/longobject.h:69: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘_PyLong_NumBits’
    /usr/include/python2.5/longobject.h:85: error: expected declaration specifiers or ‘...’ before ‘size_t’
    /usr/include/python2.5/longobject.h:108: error: expected declaration specifiers or ‘...’ before ‘size_t’
    In file included from /usr/include/python2.5/Python.h:87,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/floatobject.h:15: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:89,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/complexobject.h:39: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:92,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/stringobject.h:36: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/stringobject.h:36: error: field ‘ob_size’ declared as a function
    /usr/include/python2.5/stringobject.h:67: error: ‘PyString_Size’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:94,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/tupleobject.h:25: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/tupleobject.h:25: error: field ‘ob_size’ declared as a function
    /usr/include/python2.5/tupleobject.h:40: error: ‘PyTuple_Size’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:95,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/listobject.h:23: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/listobject.h:23: error: field ‘ob_size’ declared as a function
    /usr/include/python2.5/listobject.h:38: error: field ‘allocated’ declared as a function
    /usr/include/python2.5/listobject.h:47: error: ‘PyList_Size’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:96,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/dictobject.h:55: error: field ‘me_hash’ declared as a function
    /usr/include/python2.5/dictobject.h:71: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/dictobject.h:72: error: field ‘ma_fill’ declared as a function
    /usr/include/python2.5/dictobject.h:73: error: field ‘ma_used’ declared as a function
    /usr/include/python2.5/dictobject.h:79: error: field ‘ma_mask’ declared as a function
    /usr/include/python2.5/dictobject.h:108: error: ‘PyDict_Size’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:98,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/setobject.h:36: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/setobject.h:38: error: field ‘fill’ declared as a function
    /usr/include/python2.5/setobject.h:39: error: field ‘used’ declared as a function
    /usr/include/python2.5/setobject.h:45: error: field ‘mask’ declared as a function
    /usr/include/python2.5/setobject.h:79: error: ‘PySet_Size’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:99,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/methodobject.h:82: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:101,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/funcobject.h:22: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:102,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/classobject.h:13: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/classobject.h:24: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/classobject.h:31: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:103,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/fileobject.h:11: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/fileobject.h:12: error: expected specifier-qualifier-list before ‘FILE’
    /usr/include/python2.5/fileobject.h:38: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/fileobject.h:40: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
    /usr/include/python2.5/fileobject.h:57: error: expected declaration specifiers or ‘...’ before ‘FILE’
    /usr/include/python2.5/fileobject.h:58: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘Py_UniversalNewlineFread’
    In file included from /usr/include/python2.5/Python.h:105,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/traceback.h:13: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:106,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/sliceobject.h:23: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:107,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/cellobject.h:10: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:109,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/genobject.h:13: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:110,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/descrobject.h:46: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/descrobject.h:50: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/descrobject.h:55: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/descrobject.h:60: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/descrobject.h:65: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:111,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/weakrefobject.h:16: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/weakrefobject.h:65: error: ‘_PyWeakref_GetWeakrefCount’ declared as function returning a function
    In file included from /usr/include/python2.5/Python.h:114,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pyerrors.h:10: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/pyerrors.h:17: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/pyerrors.h:31: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/pyerrors.h:44: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/pyerrors.h:52: error: field ‘ob_refcnt’ declared as a function
    /usr/include/python2.5/pyerrors.h:329: error: expected declaration specifiers or ‘...’ before ‘size_t’
    /usr/include/python2.5/pyerrors.h:330: error: format string argument not a string type
    /usr/include/python2.5/pyerrors.h:331: error: expected declaration specifiers or ‘...’ before ‘size_t’
    In file included from /usr/include/python2.5/Python.h:118,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pyarena.h:50: error: expected declaration specifiers or ‘...’ before ‘size_t’
    In file included from /usr/include/python2.5/Python.h:120,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pythonrun.h:34: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:35: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:37: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:38: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:39: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:44: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:54: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:60: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:77: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:142: error: expected ‘)’ before ‘*’ token
    /usr/include/python2.5/pythonrun.h:144: error: expected ‘)’ before ‘*’ token
    In file included from /usr/include/python2.5/Python.h:122,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/sysmodule.h:12: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘*’ token
    In file included from /usr/include/python2.5/Python.h:124,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/import.h:33: error: expected declaration specifiers or ‘...’ before ‘size_t’
    /usr/include/python2.5/import.h:33: error: expected declaration specifiers or ‘...’ before ‘FILE’
    In file included from /usr/include/python2.5/Python.h:126,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/abstract.h:421: error: ‘PyObject_Size’ declared as function returning a function
    /usr/include/python2.5/abstract.h:433: error: ‘PyObject_Length’ declared as function returning a function
    /usr/include/python2.5/abstract.h:436: error: ‘_PyObject_LengthHint’ declared as function returning a function
    /usr/include/python2.5/abstract.h:774: error: ‘PyNumber_AsSsize_t’ declared as function returning a function
    /usr/include/python2.5/abstract.h:947: error: ‘PySequence_Size’ declared as function returning a function
    /usr/include/python2.5/abstract.h:956: error: ‘PySequence_Length’ declared as function returning a function
    /usr/include/python2.5/abstract.h:1078: error: ‘PySequence_Count’ declared as function returning a function
    /usr/include/python2.5/abstract.h:1097: error: ‘_PySequence_IterSearch’ declared as function returning a function
    /usr/include/python2.5/abstract.h:1122: error: ‘PySequence_Index’ declared as function returning a function
    /usr/include/python2.5/abstract.h:1161: error: ‘PyMapping_Size’ declared as function returning a function
    /usr/include/python2.5/abstract.h:1171: error: ‘PyMapping_Length’ declared as function returning a function
    In file included from /usr/include/python2.5/compile.h:5,
                     from /usr/include/python2.5/Python.h:128,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/code.h:11: error: field ‘ob_refcnt’ declared as a function
    In file included from /usr/include/python2.5/Python.h:131,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pystrtod.h:11: error: expected declaration specifiers or ‘...’ before ‘size_t’
    In file included from /usr/include/python2.5/Python.h:151,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pyfpe.h:129:20: error: signal.h: No such file or directory
    /usr/include/python2.5/pyfpe.h:130:20: error: setjmp.h: No such file or directory
    In file included from /usr/include/python2.5/Python.h:151,
                     from mercurial/mpatch.c:23:
    /usr/include/python2.5/pyfpe.h:132: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘PyFPE_jbuf’
    mercurial/mpatch.c:54:24: error: sys/types.h: No such file or directory
    mercurial/mpatch.c:58:25: error: arpa/inet.h: No such file or directory
    mercurial/mpatch.c:60:23: error: inttypes.h: No such file or directory
    mercurial/mpatch.c: In function ‘lalloc’:
    mercurial/mpatch.c:77: error: ‘NULL’ undeclared (first use in this function)
    mercurial/mpatch.c:77: error: (Each undeclared identifier is reported only once
    mercurial/mpatch.c:77: error: for each function it appears in.)
    mercurial/mpatch.c:82: warning: implicit declaration of function ‘malloc’
    mercurial/mpatch.c:82: warning: incompatible implicit declaration of built-in function ‘malloc’
    mercurial/mpatch.c:89: warning: implicit declaration of function ‘free’
    mercurial/mpatch.c: In function ‘combine’:
    mercurial/mpatch.c:197: error: ‘NULL’ undeclared (first use in this function)
    mercurial/mpatch.c:224: warning: implicit declaration of function ‘memcpy’
    mercurial/mpatch.c:224: warning: incompatible implicit declaration of built-in function ‘memcpy’
    mercurial/mpatch.c: In function ‘decode’:
    mercurial/mpatch.c:244: error: ‘NULL’ undeclared (first use in this function)
    mercurial/mpatch.c:249: warning: incompatible implicit declaration of built-in function ‘memcpy’
    mercurial/mpatch.c:250: warning: implicit declaration of function ‘ntohl’
    mercurial/mpatch.c:250: error: ‘uint32_t’ undeclared (first use in this function)
    mercurial/mpatch.c:250: error: expected expression before ‘)’ token
    mercurial/mpatch.c:251: error: expected expression before ‘)’ token
    mercurial/mpatch.c:252: error: expected expression before ‘)’ token
    mercurial/mpatch.c: In function ‘apply’:
    mercurial/mpatch.c:310: warning: incompatible implicit declaration of built-in function ‘memcpy’
    mercurial/mpatch.c:317: warning: incompatible implicit declaration of built-in function ‘memcpy’
    mercurial/mpatch.c: In function ‘fold’:
    mercurial/mpatch.c:330: warning: passing argument 2 of ‘PyList_GetItem’ makes pointer from integer without a cast
    mercurial/mpatch.c:332: error: ‘NULL’ undeclared (first use in this function)
    mercurial/mpatch.c:335: warning: passing argument 2 of ‘decode’ makes integer from pointer without a cast
    mercurial/mpatch.c: In function ‘patches’:
    mercurial/mpatch.c:355: error: ‘NULL’ undeclared (first use in this function)
    mercurial/mpatch.c:360: error: ‘PyObject’ has no member named ‘ob_refcnt’
    mercurial/mpatch.c:371: warning: passing argument 1 of ‘calcsize’ makes integer from pointer without a cast
    mercurial/mpatch.c:376: warning: passing argument 2 of ‘PyString_FromStringAndSize’ makes pointer from integer without a cast
    mercurial/mpatch.c:382: warning: passing argument 3 of ‘apply’ makes integer from pointer without a cast
    mercurial/mpatch.c:383: error: ‘PyObject’ has no member named ‘ob_refcnt’
    mercurial/mpatch.c:383: error: ‘PyObject’ has no member named ‘ob_type’
    mercurial/mpatch.c: In function ‘patchedsize’:
    mercurial/mpatch.c:401: error: ‘NULL’ undeclared (first use in this function)
    mercurial/mpatch.c:407: warning: incompatible implicit declaration of built-in function ‘memcpy’
    mercurial/mpatch.c:408: error: ‘uint32_t’ undeclared (first use in this function)
    mercurial/mpatch.c:408: error: expected expression before ‘)’ token
    mercurial/mpatch.c:409: error: expected expression before ‘)’ token
    mercurial/mpatch.c:410: error: expected expression before ‘)’ token
    mercurial/mpatch.c: At top level:
    mercurial/mpatch.c:435: error: ‘NULL’ undeclared here (not in a function)
    mercurial/mpatch.c: In function ‘initmpatch’:
    mercurial/mpatch.c:442: warning: passing argument 2 of ‘PyErr_NewException’ from incompatible pointer type
    mercurial/mpatch.c:442: warning: passing argument 3 of ‘PyErr_NewException’ from incompatible pointer type
    error: Setup script exited with error: command 'gcc' failed with exit status 1
    
  • Python.h error during easy_install
    If you got the following error while trying to run easy_install, it probably means you don't have the "python-dev" package installed. $ sudo apt-get install python-dev.
    Searching for mercurial
    Reading http://cheeseshop.python.org/pypi/mercurial/
    Couldn't find index page for 'mercurial' (maybe misspelled?)
    Scanning index of all packages (this may take a while)
    Reading http://cheeseshop.python.org/pypi/
    Reading http://cheeseshop.python.org/pypi/Mercurial/1.0
    Reading http://www.selenic.com/mercurial
    Best match: mercurial 1.0
    Downloading http://www.selenic.com/mercurial/release/mercurial-1.0.tar.gz
    Processing mercurial-1.0.tar.gz
    Running mercurial-1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-MfmZhU/mercurial-1.0/egg-dist-tmp-JDYH6V
    mercurial/mpatch.c:23:20: error: Python.h: No such file or directory
    mercurial/mpatch.c:64: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token
    mercurial/mpatch.c: In function 'lalloc':
    mercurial/mpatch.c:92: warning: implicit declaration of function 'PyErr_Occurred'
    mercurial/mpatch.c:93: warning: implicit declaration of function 'PyErr_NoMemory'
    mercurial/mpatch.c: In function 'decode':
    mercurial/mpatch.c:265: warning: implicit declaration of function 'PyErr_SetString'
    mercurial/mpatch.c:265: error: 'mpatch_Error' undeclared (first use in this function)
    mercurial/mpatch.c:265: error: (Each undeclared identifier is reported only once
    mercurial/mpatch.c:265: error: for each function it appears in.)
    mercurial/mpatch.c: In function 'calcsize':
    mercurial/mpatch.c:283: error: 'mpatch_Error' undeclared (first use in this function)
    mercurial/mpatch.c: In function 'apply':
    mercurial/mpatch.c:306: error: 'mpatch_Error' undeclared (first use in this function)
    mercurial/mpatch.c: At top level:
    mercurial/mpatch.c:322: error: expected ')' before '*' token
    mercurial/mpatch.c:344: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token
    mercurial/mpatch.c:392: error: expected '=', ',', ';', 'asm' or '__attribute__' before '*' token
    mercurial/mpatch.c:432: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'methods'
    mercurial/mpatch.c:439: error: expected '=', ',', ';', 'asm' or '__attribute__' before 'initmpatch'
    error: Setup script exited with error: command 'gcc' failed with exit status 1
  • Couldn't find a setup script Error:
    If you get the following error when trying to run sudo easy_install -U mercurial, it could mean that there is a "mercurial" directory in your current working directory. Change to a different directory and run the command again.
    Processing mercurial
    error: Couldn't find a setup script in mercurial
  • error: can't create or remove files in install directory
    error: can't create or remove files in install directory
    
    The following error occurred while trying to add or remove files in the
    installation directory:
    
        [Errno 13] Permission denied: '/usr/lib/python2.5/site-packages/test-easy-install-28328.write-test'
    
    The installation directory you specified (via --install-dir, --prefix, or
    the distutils default setting) was:
    
        /usr/lib/python2.5/site-packages/
    
    Perhaps your account does not have write access to this directory?  If the
    installation directory is a system-owned directory, you may need to sign in
    as the administrator or "root" account.  If you do not have administrative
    access to this machine, you may wish to choose a different installation
    directory, preferably one that is listed in your PYTHONPATH environment
    variable.
    
    For information on other options, you may wish to consult the
    documentation at:
      
      http://peak.telecommunity.com/EasyInstall.html
    
    Please make the appropriate changes for your system and try again.
    You will get this error if you don't use the sudo command. Use
    $ sudo easy_install -U mercurial

Working with files and directories in Python

I often have a difficult time finding the appropriate functions to work with files and directories in Python. I think one reason is because the Library Reference puts seemingly related functions in two different places: 11 File and Directory Access and 14.1.4 Files and Directories.

Section 11, File and Directory Access contains documentation for useful functions such as os.path.exists which checks if a path exists, glob.glob which is useful for matching filenames using the Unix-style * and ? wildcards, and shutil.copy which is similary to the Unix cp command.

This section includes a total of 11 subsections for modules related to file and directory access, but it does not contain basic commands such as os.chdir, os.listdir, or os.rename. These functions are documented in Section 14.1.4, Files and Directories, instead.

Here is a summary of some of the functions I find useful. Check the documentation for more complete and detailed information.

11 File and Directory Access
  • os.path module:
    • exists - checks if a path or file exists
      Example:
      import os.path
      print os.path.exists("c:/Windows")
      Results:
      True
    • isfile and isdir - test if the path is a file or directory, respectively.
      Example:
      import os.path
      print os.path.isfile("c:/Windows")
      print os.path.isdir("c:/Windows")
      Results:
      False
      True
    • getmtime - returns the modification time of a path
      Example:
      import os.path
      import time
      mtime = os.path.getmtime("c:/Windows")
      print time.gmtime(mtime)
      Results:
      (2008, 4, 2, 15, 58, 39, 2, 93, 0)
  • glob module:
    • glob: returns a list of paths matching a Unix-style glob pattern.
      Example:
      import glob
      print glob.glob("c:/windows/*.bmp")
      Results:
      ['c:/windows\\Blue Lace 16.bmp', 'c:/windows\\Coffee Bean.bmp', 'c:/windows\\default.bmp', 'c:/windows\\FeatherTexture.bmp', 'c:/windows\\Gone Fishing.bmp', 'c:/windows\\Greenstone.bmp', 'c:/windows\\Prairie Wind.bmp', 'c:/windows\\Rhododendron.bmp', 'c:/windows\\River Sumida.bmp', 'c:/windows\\Santa Fe Stucco.bmp', 'c:/windows\\Soap Bubbles.bmp', 'c:/windows\\winnt.bmp', 'c:/windows\\winnt256.bmp', 'c:/windows\\Zapotec.bmp']
  • shutil module:
    • copy - similar to Unix cp
    • copy2 - similar to Unix cp -p
    • copytree - similar to Unix cp -r
    • rmtree - similar to Unix rm -r
14 Generic Operating System Services - > 14.1 os --- Miscellaneous operating system interfaces -> 14.1.4 Files and Directories
  • chdir - change the current working directory
  • getcwd - return a string representing the current working directory
  • listdir - return a list of the names of the entries in the directory.
  • makedir - create a directory
  • remove - remove a file (this is identical to unlink)
  • rename - rename the file or directory
  • walk - walks a directory tree (see my os.walk example)

Install coLinux (and Ubuntu Hardy) on Win XP using Slirp to internet and TAP to host behind a corporate firewall/proxy server

coLinux allows you to run a full Linux distribution, such as Ubuntu Hardy, on top of Windows at near native speeds. It runs as a single Windows process with its own specially allocated address space. The guest linux system can run from either a separate disk partition or from a single file on your Windows filesystem.

My initial impression of coLinux was so positive, I had planned to name this post, Goodbye Cygwin, hello coLinux. After 2 months of using coLinux, I still am using Cygwin a little bit. Despite being slow, (almost unbearably slow with my company's mandatory Whole Disk Encryption), Cygwin is good at integrating the GNU tools with Windows. Currently, I am still using Cygwin to script my Windows-only project version control system and use the Cygwin version of Mercurial for my personal version control. I'm not sure if using the Linux version of Mercurial would work on Samba shares. The more I learn about Linux and the less I depend on Windows applications, the less I will need Cygwin. (Update: Mercurial 1.0 works great on Samba shares. The important thing is to setup the owner of the share to be your user id and not root. See samba config below.)

Besides that, coLinux is great. The best part for me was being able to run the dynamic, tiling, scriptable, keyboard-driven window manager, wmii. I access Windows files (and network drives) using Samba. Shell commands, emacs, ssh are all there and are so fast. GUI applications such as Firefox, OpenOffice and Gimp are available as well. However, these are slow with the slirp networking setup. With a TAP connection between coLinux and the host OS, these apps run well, however, I couldn't get the TAP connection to work in conjuction with internet access so I settled for slirp only. If I figure this out, I will update the post. Here are my install notes.

Update 4/28/2008: I got my private TAP connection to the host working along with a slirp connection for internet. This makes my X server connections fast enough to run all my GUI apps like OpenOffice,evincekpdf for pdf files, and my web browser. See this section for details.


Contents

Install coLinux
  1. Download the coLinux installer and the Ubuntu root file system image
  2. Run the coLinux Windows installer
    • Accept defaults except for:
    • Destination Folder: change to c:\coLinux
    • You do not need to install WinPCap library.
    • You will get a warning dialog saying that "TAP-Win32 Adapter V8 (coLinux)" has not passed Windows Logo testing. Click "Continue Anyway"

Configure CoLinux
I already had Cygwin running so I did this stuff in a bash shell. You can use Windows commands as well to do the same thing.
  1. Copy Ubuntu-6.06.1.ext3.1bg.bz2 to c:\coLinux. (Keep a copy of this file as a backup to revert your Ubuntu filesystem if you mess things up.)
  2. Decompress the Ubuntu file system image. (Note, this could take a minute or more depending on your system.)
    $ cd /cygdrive/c/coLinux
    $ bunzip2 Ubuntu-6.06.1.ext3.1gb.bz2
  3. Create a swap file
    $ dd if=/dev/zero of=swap_device bs=1M count=512
    $ dd if=/dev/zero of=swap_device.2gb bs=1M count=2K
    (In Windows, you can use fsutil file createnew swap_device 536870912)
  4. Create a configuration file
    $ cp example.conf colinux.conf
    Edit colinux.conf and set the following:
    cobd0="c:\coLinux\Ubuntu-6.06.1.ext3.1gb"
    cobd1="c:\coLinux\swap_device"
    mem=512
    eth0=slirp
    eth1=tuntap # only needed for TAP connection
    The following items should already be set by default:
    kernel=vmlinux
    root=/dev/cobd0
    ro
    initrd=initrd.gz

Run CoLinux
  1. Open a Windows cmd.exe shell
  2. cd c:\colinux
  3. colinux-daemon.exe -t nt @colinux.conf
    (the -t nt is optional but makes it easier to copy/paste)
  4. login as "root" with password "root"
  5. Try commands such as ls, pwd, etc.
  6. To exit, use the halt command.

Configure networking with slirp only
  1. If you use a http proxy server to get to the internet, do the following inside the running CoLinux session:
    root@ubuntu:~# echo "export http_proxy='http://your-proxy-server.com:yourportnumber'" >> .bash_profile
    root@ubuntu:~# source .bash_profile
  2. Otherwise, there is nothing to do. The Ubuntu image already has the correct /etc/network/interfaces and /etc/resolv.conf files setup.

    /etc/network/interfaces should look like this:
    # This file describes the network interfaces available on your system
    # and how to activate them. For more information, see interfaces(5).
    
    # The loopback network interface
    auto lo
    iface lo inet loopback
    
    # The primary network interface (slirp)
    auto eth0
    iface eth0 inet static
       address 10.0.2.15
       broadcast 10.0.2.255
       netmask 255.255.255.0
       gateway 10.0.2.2
    

  3. /etc/resolv.conf should look like this:
    nameserver 10.0.2.3
  4. To test internet access, use something like wget since ping doesn't work with slirp.
    root@ubuntu:~# wget www.google.com

Configure networking with slirp to internet and TAP to host
In this configuration, I set up TAP to form a private connection to the host and slirp to access the internet. If you want to use X applications, this will make the response much better than with slirp only. I had to play around with the IP addresses to find one that didn't conflict with the static IP addresses on our network. I ended up choosing "10.0.3.16" for the TAP colinux address and "10.0.3.17" for my Windows side.
  1. TAP should be installed when you install coLinux
  2. In Windows, go to "Start", "Control Panel", "Network Connections"
  3. Find the "TAP-Win32 Adapter V8 (coLinux)" adapter. For me, it was named "Local Area Connection 2". Right click on it and select "Properties".
  4. On the "General" tab, scroll down and select "Internet Protocol (TCP/IP)". Click on "Properties".
  5. Select "Use the following IP address:". Enter in "10.0.3.17" for the "IP address:" and "255.255.254.0" for the "Subnet mask:". Leave the rest blank. Click "OK" twice to finish.
  6. in coLinux, edit /etc/network/interfaces to look like the following:
    auto lo
    # Loopback interface
    iface lo inet loopback
    
    # Network interface for internet access (slirp)
    auto eth0
    iface eth0 inet static
       address 10.0.2.15
       broadcast 10.0.2.255
       netmask 255.255.255.0
       gateway 10.0.2.2
    
    # Network interface for private connection to Host OS (tap-win32)
    auto eth1
    iface eth1 inet static
       address 10.0.3.16
       network 10.0.3.0
       netmask 255.255.254.0
       broadcast 10.0.3.255
  7. Make sure your "colinux.conf" file has the following lines:
    eth0=slirp
    eth1=tuntap
  8. You will need to set your DISPLAY variable to "10.0.3.17:0"
    root@ubuntu:~# export DISPLAY=10.0.3.17:0

Update your Ubuntu Dapper installation
  1. root@ubuntu:~# apt-get update
    You can ignore the following warning:
    W: GPG error: http://security.ubuntu.com dapper-security Release: Could not exec
    ute /usr/bin/gpgv to verify signature (is gnupg installed?)
    W: GPG error: http://fr.archive.ubuntu.com dapper-updates Release: Could not exe
    cute /usr/bin/gpgv to verify signature (is gnupg installed?)
    W: GPG error: http://fr.archive.ubuntu.com dapper-backports Release: Could not e
    xecute /usr/bin/gpgv to verify signature (is gnupg installed?)
    W: GPG error: http://fr.archive.ubuntu.com dapper Release: Could not execute /us
    r/bin/gpgv to verify signature (is gnupg installed?)
    W: You may want to run apt-get update to correct these problems
  2. root@ubuntu:~# apt-get upgrade
    Answer yes to prompts.
  3. That was easy.

Run an Xterm and other apps

Warning, getting the X Window System working properly was one of the most frustrating parts of the setup for me. The behavior seemed to be even non-deterministic. This is what worked for me. You may need to consult other sources to get it working for you. I am starting to readX Power Tools to learn more about the X Window System and will update this section if I learn anything new. I used Cygwin's X server because I already had Cygwin installed. Another option is Xming.

Update 5/6/2008: After switching to the Slirp+TAP networking configuration, I've been getting errors when trying to start my X clients:

$ xeyes
No protocol specified
Error: Can't open display: 10.0.3.17:0
$ xterm
No protocol specified
xterm Xt error: Can't open display: 10.0.3.17:0
$ urxvt
No protocol specified
urxvt: can't open display 10.0.3.17:0, aborting.
To resolve this, I have been starting up a Cygwin bash shell, and running the xhost + command to disable access control. This worked, but it was annoying to do this all the time. I discovered that there is a -ac command line option to Xwin which will do the same thing. I've updated my batch file below to reflect this. Here is a list of all the Xwin command line options:
use: X [:] [option]
-a #                   mouse acceleration (pixels)
-ac                    disable access control restrictions
-audit int             set audit trail level
-auth file             select authorization file
bc                     enable bug compatibility
-br                    create root window with black background
+bs                    enable any backing store support
-bs                    disable any backing store support
-c                     turns off key-click
c #                    key-click volume (0-100)
-cc int                default color visual class
-co file               color database file
-core                  generate core dump on fatal error
-dpi int               screen resolution in dots per inch
-deferglyphs [none|all|16] defer loading of [no|all|16-bit] glyphs
-f #                   bell base (0-100)
-fc string             cursor font
-fn string             default font name
-fp string             default font path
-help                  prints message with these options
-I                     ignore all remaining arguments
-ld int                limit data space to N Kb
-lf int                limit number of open files to N
-ls int                limit stack space to N Kb
-logo                  enable logo in screen saver
nologo                 disable logo in screen saver
-nolisten string       don't listen on protocol
-noreset               don't reset after last client exists
-reset                 reset after last client exists
-p #                   screen-saver pattern duration (minutes)
-pn                    accept failure to listen on all ports
-nopn                  reject failure to listen on all ports
-r                     turns off auto-repeat
r                      turns on auto-repeat 
-render [default|mono|gray|color] set render color alloc policy
-s #                   screen-saver timeout (minutes)
-sp file               security policy file
-su                    disable any save under support
-t #                   mouse threshold (pixels)
-terminate             terminate at server reset
-to #                  connection time out
-tst                   disable testing extensions
ttyxx                  server started from init on /dev/ttyxx
v                      video blanking for screen-saver
-v                     screen-saver without video blanking
-wm                    WhenMapped default backing-store
-x string              loads named extension at init time 
-maxbigreqsize         set maximal bigrequest size 
+extension name        Enable extension
-extension name        Disable extension
-query host-name       contact named host for XDMCP
-broadcast             broadcast for XDMCP
-indirect host-name    contact named host for indirect XDMCP
-port port-num         UDP port number to send messages to
-from local-address    specify the local address to connect from
-once                  Terminate server after one session
-class display-class   specify display class to send in manage
-cookie xdm-auth-bits  specify the magic cookie for XDMCP
-displayID display-id  manufacturer display ID for request
The X Keyboard Extension adds the following arguments:
-kb                    disable the X Keyboard Extension
+kb                    enable the X Keyboard Extension
[+-]accessx [ timeout [ timeout_mask [ feedback [ options_mask] ] ] ]
                       enable/disable accessx key sequences
-ar1                   set XKB autorepeat delay
-ar2                   set XKB autorepeat interval
-noloadxkb             don't load XKB keymap description
-xkbdb                 file that contains default XKB keymaps
-xkbmap                XKB keyboard description to load on startup
-depth bits_per_pixel
 Specify an optional bitdepth to use in fullscreen mode
 with a DirectDraw engine.
-emulate3buttons [timeout]
 Emulate 3 button mouse with an optional timeout in
 milliseconds.
-engine engine_type_id
 Override the server's automatically selected engine type:
  1 - Shadow GDI
  2 - Shadow DirectDraw
  4 - Shadow DirectDraw4 Non-Locking
-fullscreen
 Run the server in fullscreen mode.
-refresh rate_in_Hz
 Specify an optional refresh rate to use in fullscreen mode
 with a DirectDraw engine.
-screen scr_num [width height [x y] | [[WxH[+X+Y]][@m]] ]
 Enable screen scr_num and optionally specify a width and
 height and initial position for that screen. Additionally
 a monitor number can be specified to start the server on,
 at which point, all coordinates become relative to that
 monitor (Not for Windows NT4 and 95). Examples:
  -screen 0 800x600+100+100@2 ; 2nd monitor offset 100,100 size 800x600
  -screen 0 1024x768@3        ; 3rd monitor size 1024x768
  -screen 0 @1 ; on 1st monitor using its full resolution (the default)
-lesspointer
 Hide the windows mouse pointer when it is over an inactive
 Cygwin/X window.  This prevents ghost cursors appearing where
 the Windows cursor is drawn overtop of the X cursor
-nodecoration
 Do not draw a window border, title bar, etc.  Windowed
 mode only.
-mwextwm
 Run the server in multi-window external window manager mode.
-internalwm
 Run the internal window manager.
-rootless
 Run the server in rootless mode.
-multiwindow
 Run the server in multi-window mode.
-multiplemonitors
 EXPERIMENTAL: Use the entire virtual screen if multiple
 monitors are present.
-clipboard
 Run the clipboard integration module.
 Do not use at the same time as 'xwinclip'.
-nounicodeclipboard
 Do not use Unicode clipboard even if NT-based platform.
-scrollbars
 In windowed mode, allow screens bigger than the Windows desktop.
 Moreover, if the window has decorations, one can now resize
 it.
-[no]trayicon
 Do not create a tray icon.  Default is to create one
 icon per screen.  You can globally disable tray icons with
 -notrayicon, then enable it for specific screens with
 -trayicon for those screens.
-clipupdates num_boxes
 Use a clipping region to constrain shadow update blits to
 the updated region when num_boxes, or more, are in the
 updated region.  Currently supported only by `-engine 1'.
-[no]unixkill
 Ctrl+Alt+Backspace exits the X Server.
-[no]winkill
 Alt+F4 exits the X Server.
-xkbrules XKBRules
 Equivalent to XKBRules in XF86Config files.
-xkbmodel XKBModel
 Equivalent to XKBModel in XF86Config files.
-xkblayout XKBLayout
 Equivalent to XKBLayout in XF86Config files.
 For example: -xkblayout de
-xkbvariant XKBVariant
 Equivalent to XKBVariant in XF86Config files.
 For example: -xkbvariant nodeadkeys
-xkboptions XKBOptions
 Equivalent to XKBOptions in XF86Config files.
-logfile filename
 Write logmessages to  instead of /tmp/Xwin.log.
-logverbose verbosity
 Set the verbosity of logmessages. [NOTE: Only a few messages
 respect the settings yet]
  0 - only print fatal error.
  1 - print additional configuration information.
  2 - print additional runtime information [default].
  3 - print debugging and tracing information.
-[no]keyhook
 Grab special windows key combinations like Alt-Tab or the Menu key.
 These keys are discarded by default.
-swcursor
 Disable the usage of the windows cursor and use the X11 software cursor instead
You can view this list by typing Xwin --help in a Cygwin bash shell. Also, in the process, I tried the Xming X server. However, I couldn't get it to work with wmii, so I decided to stick with Cygwin's X server.

  1. Edit your startxwin.bat file (located in "C:\cygwin\usr\X11R6\bin") so that it starts X in rootless mode, with clipboard support, disables access control, and doesn't run anything else. Run the new bat file.
    @echo off
    SET DISPLAY=127.0.0.1:0.0
    SET CYGWIN_ROOT=\cygwin
    SET RUN=%CYGWIN_ROOT%\bin\run -p /usr/X11R6/bin
    SET PATH=.;%CYGWIN_ROOT%\bin;%CYGWIN_ROOT%\usr\X11R6\bin;%PATH%
    SET XAPPLRESDIR=/usr/X11R6/lib/X11/app-defaults
    SET XCMSDB=/usr/X11R6/lib/X11/Xcms.txt
    SET XKEYSYMDB=/usr/X11R6/lib/X11/XKeysymDB
    SET XNLSPATH=/usr/X11R6/lib/X11/locale
    if not exist %CYGWIN_ROOT%\tmp\.X11-unix\X0 goto CLEANUP-FINISH
    attrib -s %CYGWIN_ROOT%\tmp\.X11-unix\X0
    del %CYGWIN_ROOT%\tmp\.X11-unix\X0
    :CLEANUP-FINISH
    if exist %CYGWIN_ROOT%\tmp\.X11-unix rmdir %CYGWIN_ROOT%\tmp\.X11-unix
    
    %RUN% XWin -clipboard -silent-dup-error -rootless -ac
  2. If you are using Slirp-only networking:
    • Obtain your Windows host IP address by opening a cmd.exe shell and running ipconfig
    • In coLinux, set your DISPLAY variable to this IP address:
      root@ubuntu:~# export DISPLAY=xxx.xxx.xxx.xxx:0.0
      where xxx.xxx.xxx.xxx is your IP address. Don't forget the :0.0 at the end.
  3. If you are using TAP plus Slirp configuration described above, set your DISPLAY variable to "10.0.3.17:0"
    root@ubuntu:~# export DISPLAY=10.0.3.17:0
  4. root@ubuntu:~# apt-get install xterm
    Answer yes to prompts.
  5. root@ubuntu:~# xterm&
    Xterm comes up!
  6. You can now install and run other high level apps like firefox or emacs

Expand the root file system image
You will need to expand the size your root filesystem image to hold the larger Ubuntu installation. I expanded it to 4GB for Xubuntu Hardy and am currently using about 87% of that. You might want to expand it to 5GB or more especially if you will be installing a lot of apps. See http://colinux.wikia.com/wiki/ExpandingRoot for more information. Note, Google's calculator function can be useful for calculating block sizes. (e.g. entering in 4GB / 16KB into the Google search box returns: (4 gigabytes) / (16 kilobytes) = 262 144)
  1. If you have coLinux running, shut it down, with halt
  2. In a Cygwin bash shell, make a copy of your current image.
    $ cp Ubuntu-6.06.1.ext3.1gb Ubuntu-6.06.1.ext3.4gb
    This may take a while.
  3. Expand the new file:
    $ dd bs=16384 if=/dev/zero of=Ubuntu-6.06.1.ext3.4gb seek=65536 count=262144

    bs=16384 sets the blocksize to 16KB
    seek=65536 skips the first 65536 blocks (1GB/16KB = 65536)
    count=262144 adds 262144 blocks to the image (4GB/16KB = 262144)

    This may take a while also. It should display the following when complete:
    262144+0 records in
    262144+0 records out
    4294967296 bytes (4.3 GB) copied, 154.362 s, 27.8 MB/s
  4. Change your colinux.conf file to use the new file system image:
    cobd0="c:\coLinux\Ubuntu-6.06.1.ext3.4gb"
  5. Switch back to the cmd.exe console, and start coLinux again
  6. root@ubuntu:~# apt-get install ext2resize
  7. root@ubuntu:~# ext2online /dev/cobd0
  8. Look at your available space:
    root@ubuntu:~# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/cobd0             4128448    411148   3507624  11% /
    varrun                  128096        36    128060   1% /var/run
    varlock                 128096         0    128096   0% /var/lock
    devshm                  128096         0    128096   0% /dev/shm
  9. Note, you may want to make a backup copy of your new expanded root file system image at this point in case things get hosed up later on. Be sure to halt coLinux first. Note, this will take even longer than the 1gb filesystem copy.

Upgrade to Ubuntu Hardy
Update 4/28/2008: When I first installed coLinux, Ubuntu Gutsy was the latest release. Because there was no direct upgrade path from Ubuntu 6.06 Dapper to Ubuntu 7.10 Gutsy, I had to incrementally upgrade from Dapper to Edgy, Edgy to Feisty, and Feisty to Gutsy. Later, when Hardy came out, I upgraded from Gutsy to Hardy. If you are upgrading to Dapper to Hardy for the first time, you can use the incremental method, or the direct method. Ubuntu supports a direct upgrade from 6.06 Dapper Drake to 8.04 Hardy Heron because both are Long Term Support (LTS) releases. See the Ubuntu upgrade documentation for more information. Note, I did not test the direct upgrade method.
  1. In coLinux:
    root@ubuntu:~# apt-get update
  2. root@ubuntu:~# apt-get install ubuntu-minimal
  3. root@ubuntu:~# apt-get install ubuntu-standard
  4. root@ubuntu:~# apt-get install xubuntu-desktop
  5. Edit your sources.list file and change "dapper" to "edgy"
    root@ubuntu:~# nano /etc/apt/sources.list
  6. root@ubuntu:~# apt-get update
  7. root@ubuntu:~# apt-get upgrade
  8. root@ubuntu:~# apt-get dist-upgrade
  9. Change "edgy" to "feisty" in /etc/apt/sources.list and repeat steps 6-8.
  10. Change "feisty" to "gutsy" in /etc/apt/sources.list and repeat steps 6-8.
  11. Change "gutsy" to "hardy" in /etc/apt/sources.list and repeat steps 6-8.
  12. Remove unneeded packages:
    root@ubuntu:~# apt-get autoremove

Disable some startup services
This is optional but may give you fewer warnings. In coLinux, go to the /etc/rc2.d directory and follow the instructions in the README file for disabling services. Here are some suggested services to disable:
  • acpid, acpi-support, apmd: used for power management
  • bluetooth: used for bluetooth
  • hotkey-setup: used to configure laptop hotkeys
  • pcmciautils: tools for PCMCIA cards (PC cards)
  • powernowd: controls CPU speed and voltage using the sysfs interface
  • vbesave: used to save the video card state

Configure Ubuntu
  1. Don't start gdm (the display manager which starts the X server)
    root@ubuntu:~# mv /etc/rc2.d/S13gdm /etc/rc2.d/K87gdm
  2. Set the timezone
    Start your Cygwin X server if it is not already running.
    root@ubuntu:~# time-admin &
    Set your timezone.
  3. Set the root password
    root@ubuntu:~# passwd
  4. Set local machine name:
    root@ubuntu:~# nano /etc/hostname
    root@ubuntu:~# nano /etc/hosts
  5. Create a user account
    root@ubuntu:~# adduser sofeng
    Note, adduser is a higher level command than useradd. adduser will set up a group and home directory.
  6. Set permissions on /dev/tty* (Note: not sure if this is needed)
    root@ubuntu:~# chgrp tty /dev/tty*
    Edit /etc/group to have the following line:
    tty:x:5:sofeng
  7. Make a pty device
    root@ubuntu:~# cd /dev
    root@ubuntu:~# MAKEDEV pty
  8. Add environment variables to .bash_profile
    Use the printenv command to list your environment variables.

Create a home filesystem image
  1. Create a 10GB blank file. In Cygwin,
    $ dd if=/dev/zero of=home-fs-image.10gb bs=1M count=10K
  2. Add the following to your c:\coLinux\colinux.conf file:
    cobd2="c:\coLinux\home-fs-image.10gb"
  3. Restart coLinux, then do the following:
    Create an ext2 file system on the new disk image
    root@ubuntu:~# mke2fs -F /dev/cobd2
    Turn this into an ext3 filesystem
    root@ubuntu:~# tune2fs -i 0 -j /dev/cobd4
  4. Mount the new filesystem to /home
    root@ubuntu:~# mv /home /home_backup
    root@ubuntu:~# mkdir /home
    root@ubuntu:~# mount /dev/cobd2 /home
    Check it worked:
    root@ubuntu:~# df
  5. To mount the filesystem automatically at startup, add the following line to your /etc/fstab:
    /dev/cobd2   /home   ext3   defaults   0 0

Setup the swap file
  1. Add the following to /etc/fstab:
    /dev/cobd1   swap   swap   defaults  0 0
  2. Initialize the swap partition
    root@ubuntu:~# sudo mkswap /dev/cobd1
  3. make use of swap now, without rebooting system:
    root@ubuntu:~# swapon -a
  4. Check that swap total is non-zero:
    root@ubuntu:~# free

Mount Windows filesystem as Samba share
  1. Setup Windows networking:
    • In Windows Explorer, right click on your C: drive and select "Properties"
    • Click on the "Sharing" tab
    • Select "Share this folder". Set the "Share name:" to "E$". Setup "Permissions" as necessary.
  2. create the location for your samba share:
    $ sudo mkdir -p /mnt/cdrive
  3. add the following to /etc/fstab:
    //yourcomputer.example.com/E$ /mnt/cdrive smbfs auto,user,rw,dir_mode=0777,file_mode=0777,uid=sofeng,gid=sofeng,credentials=/etc/smb-credentials 0 0
    /mnt/cdrive is the mount point
    smbfs means it is a samba filesystem type
    auto means it will be mounted at startup
    user means any user can mount it
    rw means it will be readable and writable
    file_mode=0777 specifies that files will be readable, writable, and executable by everyone (this replaces the deprecated fmask)
    dir_mode=0777 specifies that directories will be readable, writable, and executable by everyone (this replaces the deprecated dmask)
    uid=sofeng means the owner is sofeng
    gid=sofeng means the group is sofeng
    credentials=/etc/smb-credentials specifies the file that contains the user name and password
  4. create /etc/smb-credentials
    username=yourdomain\yourusername
    password=yourpassword
    $ sudo chmod 600 /etc/smb-credentails
  5. Mount the drive:
    $ sudo mount /mnt/cdrive

Set up wmii window manager
  1. Install wmii window manager
    apt-get install wmii
  2. I use emacs so the following steps are used to change the mod key.
  3. Download KeyTweak for Windows and remap your "Left Windows" key to "Context Menu".
  4. Use xmodmap to setup the "Menu" key to be "mod4". For my keyboard, my ~/.Xmodmap file looks like this:
    clear mod4
    keycode 115 =
    keycode 127 =
    keycode 117 = Super_L
    add mod4 = Super_L
    Then run xmodmap in one of your startup scripts (.xinitrc, .bash_profile, etc.)
    xmodmap ~/.Xmodmap

Backup on Linux: rsnapshot vs. rdiff-backup (vs. Time Machine)

Apple's release of Leopard and the included backup utility, Time Machine, has generated a lot of talk about backups recently. I will admit Time Machine is pretty cool and believe that it is a bit more than a glorified GUI on top of an existing *nix tool as some have claimed. However, the core functionality is very similar to the command-line tool, rsnapshot, which is itself based on a rsync script by Mike Rubel. Time Machine added a couple of features and a GUI to make it easy to use. Since I prefer the command line over GUIs most of the time anyways, rsnapshot seemed perfect for me.

To be complete, I researched a number of other backup utilities for Linux. Dirvish and flyback were out because I prefer the command line and they didn't seem to offer anything more than rsnapshot. Scripting rsync wouldn't get me anything more than rsnapshot either, plus it would be more work. In the end, I eliminated all but rsnapshot and another command line tool called rdiff-backup. Rdiff-backup has some advantages over rsnapshot (and Time Machine) because it stores compressed deltas of each version of a file instead of a complete copy of the file each time it changes. This is not a big deal for small files, but for large files that change often, it makes a significant difference. However, the big disadvantage of rdiff-backup, for me, was the inablility to perform different levels of backup, such as hourly, daily, weekly, monthly, etc. Depending on the needs of the user, this could negate the space saving advantage by needing to keep a large number of snapshots.

I ended up choosing rsnapshot over rdiff-backup because of this last reason. It seems rdiff-backup is closer to a version control tool and rsnapshot closer to a traditional backup solution. It would be great to create a hybrid of the two tools to gain the advantages of each. I started to dig into the source of rdiff-backup (python source makes me happy), but I didn't want to get too sidetracked with another project. For now, I am using Mercurial to version control my /etc and /home/.* config files and rsnapshot as a broader, general purpose backup tool.


Here is my comparison of rsnapshot and rdiff-backup:

Similar Features
  • both use an rsync-like algorithm to transfer data (rsnapshot actually uses rsync; rdiff-backup uses the python librsync library)
  • both can be used over ssh (though rsnapshot cannot push over ssh without some extra scripting)
  • both use a simple copy of the source for the current backup
Written inrsnapshot is written in Perl; rdiff-backup is written in Python and C
Sizerdiff-backup stores previous versions as compressed deltas to the current version similar to a version control system. rsnapshot uses actual files and hardlinks to save space. For small files, storage size is similar. For large files that change often, such as logfiles, databases, etc., rdiff-backup requires significantly less space for a given number of versions.
Speedrdiff-backup is slower than rsnapshot
Metadatardiff-backup stores file metadata, such as ownership, permissions, and dates, separately.
TransparencyFor rsnapshot, all versions of the backup are accessible as plain files. For rdiff-backup, only the current backup is accessible as plain files. Previous versions are stored as rdiff deltas.
Backup levelsrsnapshot supports multiple levels of backup such as monthly, weekly, and daily. rdiff-backup can only delete snapshots earlier than a given date; it cannot delete snapshots in between two dates.
CommunityBased on the number of responses to my post on the mailing lists (rsnapshot: 6, rdiff-backup: 0), rsnapshot has a more active community.

Reference: Chapter 7 in Backup & Recovery O'Reilly 2007

How to get a summary of disk usage with du

The following command gives me a listing of the sizes of all the directories in my root directory.

$ sudo du -chs /*

Here is the results of that command on my new (1 day old) Dell 530N Ubuntu desktop: (Note, the 5.0GB for the /home directory is due to the 5.0GB ubuntu-dell-reinstall.iso file. The rest of my /home directory is only 18MB.)

4.8M    /bin
18M     /boot
0       /cdrom
128K    /dev
9.1M    /etc
5.0G    /home
4.0K    /initrd
0       /initrd.img
154M    /lib
16K     /lost+found
12K     /media
4.0K    /mnt
13M     /opt
0       /proc
740K    /root
6.3M    /sbin
4.0K    /srv
0       /sys
80K     /tmp
2.1G    /usr
317M    /var
0       /vmlinuz
7.6G    total

To determine disk usage in a different directory, cd to that directory and run sudo du -chs *

Undoing changes by pppoeconf

I just got my new Dell with Ubuntu and thought I needed to configure something to get my new DSL service up and running so I ran pppoeconf. (It turns out I needed to register with ATT first. See my previous post.) So I needed to undo the changes I made with pppoeconf. Googling around a little, it appears there is no undo command for pppoeconf. However, I found the key configuration file was: /etc/network/interfaces. I looked at it, but wasn't sure which stuff to delete. I popped in my Ubuntu live installation disk (included with my Dell). Using the live CD, the internet worked great. So I looked at the /etc/network/interfaces while using the live CD and then edited my installed version to match.


Here is what my /etc/network/interfaces file looked like after running pppoeconf:

auto lo
iface lo inet loopback


auto dsl-provider
iface dsl-provider inet ppp
pre-up /sbin/ifconfig eth0 up # line maintained by pppoeconf
provider dsl-provider

auto eth0
iface eth0 inet manual

Here is the default /etc/network/interfaces on Ubuntu 7.10 Gutsy (only two lines):

auto lo
iface lo inet loopback

Setting up ATT Yahoo DSL with Ubuntu linux

I ordered new ATT/Yahoo DSL service the day before ordering my Dell 530N desktop with Ubuntu 7.10 Gutsy installed. ATT told me only Windows and Mac were supported for their DSL service. I guess they have an install CD that only runs on Windows or Mac. After some internet searching, I found that it is possible to set up the DSL service without the install CD. Here is the link which helped me with the process. The following is the summary of my steps on Ubuntu linux. I am using a SpeedStream DSL modem; this may or may not work with other modems.

  1. Connect the modem to everything (phone line to phone jack, ethernet cable to your PC, and, of course, the power adapter).
  2. Turn on the modem.
  3. On the underside of my modem was a yellow sticker that said "For Advanced Modem Configuration Go to: http://192.169.0.1"
  4. Go to http://192.169.0.1 in your web browser
  5. Enter the following login information:
  6. Enter in the Modem Access Code found on the bottom of your DSL modem.
  7. It will look like you are connected to the internet at this point, but you won't be able to get to Yahoo or Google. This login provides limited connectivity so you can get to the ATT registration site.
  8. Go to https://sbcreg.sbcglobal.net to get to the registration site. (If you are part of another domain, you will need to go to a different registration site. e.g. att.net customers go to http://helpme.att.net/register)
  9. Complete the registration process. You will need your DSL phone number. When you get to the Member ID and password confirmation page, you are done. Print/write down/memorize this information. You do not need to complete the rest of the registration process.
  10. Go back to http://192.169.0.1 in your web browser and enter in your new Member ID and password. (You will probably have to logout and then login again.)
  11. You should now be connected to the internet.

Note: this assumes the default Ubuntu 7.10 network configuration which automatically detects your active DSL connection. If you ran pppoeconf by mistake, see this post.