SaltyCrane Blog — Notes on JavaScript and web development

PyQt 4.3 QTableView / QAbstractTableModel formatting example

This example displays the output of the "dir" command using QTableView and QAbstractTableModel. It adds some more formatting to my previous example such as specifying the font, a header, column width, row height, etc.

import re
import os
import sys 
from PyQt4.QtCore import * 
from PyQt4.QtGui import * 
 
def main(): 
    app = QApplication(sys.argv) 
    w = MyWindow() 
    w.show() 
    sys.exit(app.exec_()) 
 
class MyWindow(QWidget): 
    def __init__(self, *args): 
        QWidget.__init__(self, *args) 

        # create table
        self.get_table_data()
        table = self.createTable() 
         
        # layout
        layout = QVBoxLayout()
        layout.addWidget(table) 
        self.setLayout(layout) 

    def get_table_data(self):
        stdouterr = os.popen4("dir c:\\")[1].read()
        lines = stdouterr.splitlines()
        lines = lines[5:]
        lines = lines[:-2]
        self.tabledata = [re.split(r"\s+", line, 4)
                     for line in lines]

    def createTable(self):
        # create the view
        tv = QTableView()

        # set the table model
        header = ['date', 'time', '', 'size', 'filename']
        tm = MyTableModel(self.tabledata, header, self) 
        tv.setModel(tm)

        # set the minimum size
        self.setMinimumSize(400, 300)

        # hide grid
        tv.setShowGrid(False)

        # set the font
        font = QFont("Courier New", 8)
        tv.setFont(font)

        # hide vertical header
        vh = tv.verticalHeader()
        vh.setVisible(False)

        # set horizontal header properties
        hh = tv.horizontalHeader()
        hh.setStretchLastSection(True)

        # set column width to fit contents
        tv.resizeColumnsToContents()

        # set row height
        nrows = len(self.tabledata)
        for row in xrange(nrows):
            tv.setRowHeight(row, 18)

        # enable sorting
        # this doesn't work
        #tv.setSortingEnabled(True)

        return tv
 
class MyTableModel(QAbstractTableModel): 
    def __init__(self, datain, headerdata, parent=None, *args): 
        QAbstractTableModel.__init__(self, parent, *args) 
        self.arraydata = datain
        self.headerdata = headerdata
 
    def rowCount(self, parent): 
        return len(self.arraydata) 
 
    def columnCount(self, parent): 
        return len(self.arraydata[0]) 
 
    def data(self, index, role): 
        if not index.isValid(): 
            return QVariant() 
        elif role != Qt.DisplayRole: 
            return QVariant() 
        return QVariant(self.arraydata[index.row()][index.column()]) 

    def headerData(self, col, orientation, role):
        if orientation == Qt.Horizontal and role == Qt.DisplayRole:
            return QVariant(self.headerdata[col])
        return QVariant()

if __name__ == "__main__": 
    main()

PyQt example: How to run a command and disply its stdout

This widget consists of a QLineEdit class and a QTextEdit class. The user enters a DOS command in the input box, hits RETURN, and the stdout from the command is displayed in the text box.

import os
import sys 
from PyQt4.QtCore import * 
from PyQt4.QtGui import * 
 
def main(): 
    app = QApplication(sys.argv) 
    w = MyWindow() 
    w.show() 
    sys.exit(app.exec_()) 
 
class MyWindow(QWidget): 
    def __init__(self, *args): 
        QWidget.__init__(self, *args) 
 
        # create objects
        label = QLabel(self.tr("Enter command and press Return"))
        self.le = QLineEdit()
        self.te = QTextEdit()

        # layout
        layout = QVBoxLayout(self)
        layout.addWidget(label)
        layout.addWidget(self.le)
        layout.addWidget(self.te)
        self.setLayout(layout) 

        # create connection
        self.connect(self.le, SIGNAL("returnPressed(void)"),
                     self.run_command)

    def run_command(self):
        cmd = str(self.le.text())
        stdouterr = os.popen4(cmd)[1].read()
        self.te.setText(stdouterr)
  
if __name__ == "__main__": 
    main()

How to display the stdout of a command with PyQt

This widget contains a QPushButton and a QTextEdit box. When the button is pushed, the results of the dir is displayed in the text box.

import os
import sys 
from PyQt4.QtCore import * 
from PyQt4.QtGui import * 
 
def main(): 
    app = QApplication(sys.argv) 
    w = MyWindow() 
    w.show() 
    sys.exit(app.exec_()) 
 
class MyWindow(QWidget): 
    def __init__(self, *args): 
        QWidget.__init__(self, *args) 
 
        # create objects
        self.pb = QPushButton(self.tr("Run command"))
        self.te = QTextEdit()

        # layout
        layout = QVBoxLayout(self)
        layout.addWidget(self.pb)
        layout.addWidget(self.te)
        self.setLayout(layout) 

        # create connection
        self.connect(self.pb, SIGNAL("clicked(bool)"),
                     self.run_command)

    def run_command(self):
        stdouterr = os.popen4("dir")[1].read()
        self.te.setText(stdouterr)
  
if __name__ == "__main__": 
    main()

How to copy Python lists or other objects

This problem had me stumped for a while today. If I have a list a, setting b = a doesn't make a copy of the list a. Instead, it makes a new reference to a. For example, see the interactive Python session below:

Python 2.5.1 (r251:54863, May 18 2007, 16:56:43)
[GCC 3.4.4 (cygming special, gdc 0.12, using dmd 0.125)] on cygwin
Type "help", "copyright", "credits" or "license" for more information.
>>> a = [1,2,3]
>>> b = a
>>> b
[1, 2, 3]
>>> a.append(4)
>>> a
[1, 2, 3, 4]
>>> b
[1, 2, 3, 4]
>>> 

Here is a quick reference extracted from Chapter 9 in Learning Python, 1st Edition.

To make a copy of a list, use the following:
newList = myList[:]
newList2 = list(myList2)         # alternate method

To make a copy of a dict, use the following:
newDict = myDict.copy()

To make a copy of some other object, use the copy module:
import copy
newObj = copy.copy(myObj)        # shallow copy
newObj2 = copy.deepcopy(myObj2)  # deep copy

For more information on shallow and deep copies with the copy module, see the Python docs.

Python circular buffer

Here is a simple circular buffer, or ring buffer, implementation in Python. It is a first-in, first-out (FIFO) buffer with a fixed size.

class RingBuffer:
    def __init__(self, size):
        self.data = [None for i in xrange(size)]

    def append(self, x):
        self.data.pop(0)
        self.data.append(x)

    def get(self):
        return self.data
Here is an example where the buffer size is 4. Ten integers, 0-9, are inserted, one at a time, at the end of the buffer. Each iteration, the first element is removed from the front of the buffer.
buf = RingBuffer(4)
for i in xrange(10):
    buf.append(i)
    print buf.get()

Here are the results:
[None, None, None, 0]
[None, None, 0, 1]
[None, 0, 1, 2]
[0, 1, 2, 3]
[1, 2, 3, 4]
[2, 3, 4, 5]
[3, 4, 5, 6]
[4, 5, 6, 7]
[5, 6, 7, 8]
[6, 7, 8, 9]

References:

Django project #3: Creating models

This section in the tutorial was actually very straightforward. Here is a record of what I did. I first created a polls app.

sofeng@tortoise:~/Web/mysite$ python manage.py startapp polls
sofeng@tortoise:~/Web/mysite$ ll
total 92
-rw-r--r-- 1     0 2007 11/09 21:42 __init__.py
-rw-r--r-- 1   131 2007 11/09 21:56 __init__.pyc
-rwxr-xr-x 1   542 2007 11/09 21:42 manage.py
-rw-r--r-- 1 35840 2007 11/28 23:07 mydb
-rw-r--r-- 1 25600 2007 11/14 23:13 mydb_backup
drwxr-xr-x 2  4096 2007 11/28 23:15 polls
-rw-r--r-- 1  2886 2007 11/28 22:41 settings.py
-rw-r--r-- 1  1873 2007 11/28 22:42 settings.pyc
-rw-r--r-- 1   224 2007 11/28 22:44 urls.py
-rw-r--r-- 1   302 2007 11/28 22:45 urls.pyc
sofeng@tortoise:~/Web/mysite$ ll polls
total 8
-rw-r--r-- 1  0 2007 11/28 23:15 __init__.py
-rw-r--r-- 1 57 2007 11/28 23:15 models.py
-rw-r--r-- 1 26 2007 11/28 23:15 views.py
I entered the following into polls/models.py
from django.db import models

class Poll(models.Model):
    question = models.CharField(maxlength=200)
    pub_date = models.DateTimeField('date published')

class Choice(models.Model):
    poll = models.ForeignKey(Poll)
    choice = models.CharField(maxlength=200)
    votes = models.IntegerField()
I edited settings.py to include the new app:
INSTALLED_APPS = (
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    'mysite.polls',
)
I ran python manage.py sql polls to see the SQL CREATE TABLE statements for the polls app.
sofeng@tortoise:~/Web/mysite$ python manage.py sql polls
BEGIN;
CREATE TABLE "polls_poll" (
    "id" integer NOT NULL PRIMARY KEY,
    "question" varchar(200) NOT NULL,
    "pub_date" datetime NOT NULL
);
CREATE TABLE "polls_choice" (
    "id" integer NOT NULL PRIMARY KEY,
    "poll_id" integer NOT NULL REFERENCES "polls_poll" ("id"),
    "choice" varchar(200) NOT NULL,
    "votes" integer NOT NULL
);
COMMIT;
I ran python manage.py syncdb to create the model tables in the database.
sofeng@tortoise:~/Web/mysite$ python manage.py syncdb
Creating table polls_poll
Creating table polls_choice
Installing index for polls.Choice model
Loading 'initial_data' fixtures...
No fixtures found.
I played with the shell.
sofeng@tortoise:~/Web/mysite$ python manage.py shell
Python 2.5.1 (r251:54863, Oct  5 2007, 13:36:32) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from mysite.polls.models import Poll, Choice
>>> Poll.objects.all()
[]
>>> from datetime import datetime
>>> p = Poll(question="What's up?", pub_date=datetime.now())
>>> p.save()
>>> p.id
1
>>> p.question
"What's up?"
>>> p.pub_date
datetime.datetime(2007, 11, 29, 1, 34, 4, 883118)
>>> p.pub_date = datetime(2005, 4, 1, 0, 0)
>>> p.save()
>>> Poll.objects.all()
[]
>>> 
I edited polls/models.py so that it looked like this:
import datetime
from django.db import models

class Poll(models.Model):
    question = models.CharField(maxlength=200)
    pub_date = models.DateTimeField('date published')

    def __str__(self):
        return self.question

    def was_published_today(self):
        return self.pub_date.date() == datetime.date.today()

class Choice(models.Model):
    poll = models.ForeignKey(Poll)
    choice = models.CharField(maxlength=200)
    votes = models.IntegerField()

    def __str__(self):
        return self.choice
I went back to the shell.
sofeng@tortoise:~/Web/mysite$ python manage.py shell
Python 2.5.1 (r251:54863, Oct  5 2007, 13:36:32) 
[GCC 4.1.3 20070929 (prerelease) (Ubuntu 4.1.2-16ubuntu2)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from mysite.polls.models import Poll, Choice
>>> Poll.objects.all()
[<Poll: What's up?>]
>>> Poll.objects.filter(id=1)
[<Poll: What's up?>]
>>> Poll.objects.filter(question__startswith='What')
[<Poll: What's up?>]
>>> Poll.objects.get(pub_date__year=2005)
<Poll: What's up?>
>>> Poll.objects.get(id=2)
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/var/lib/python-support/python2.5/django/db/models/manager.py", line 73, in get
    return self.get_query_set().get(*args, **kwargs)
  File "/var/lib/python-support/python2.5/django/db/models/query.py", line 252, in get
    raise self.model.DoesNotExist, "%s matching query does not exist." % self.model._meta.object_name
DoesNotExist: Poll matching query does not exist.
>>> Poll.objects.get(pk=1)
<Poll: What's up?>
>>> p = Poll.objects.get(pk=1)
>>> p.was_published_today()
False
>>> p = Poll.objects.get(pk=1)
>>> p.choice_set.create(choice='Not much', votes=0)
<Choice: Not much>
>>> p.choice_set.create(choice='The sky', votes=0)
<Choice: The sky>
>>> c = p.choice_set.create(choice='Just hacking again', votes=0)
>>> c.poll
<Poll: What's up?>
>>> p.choice_set.all()
[<Choice: Not much>, <Choice: The sky>, <Choice: Just hacking again>]
>>> p.choice_set.count()
3
>>> Choice.objects.filter(poll__pub_date__year=2005)
[<Choice: Not much>, <Choice: The sky>, <Choice: Just hacking again>]
>>> c = p.choice_set.filter(choice__startswith='Just hacking')
>>> c.delete()
>>> 
That's it for now. Next time I will work with Django's admin web interface.

How to remove C style comments using Python

The Perl FAQ has an entry How do I use a regular expression to strip C style comments from a file? Since I've switched to Python, I've adapted the Perl solution to Python. This regular expression was created by Jeffrey Friedl and later modified by Fred Curtis. I'm not certain, but it appears to use the "unrolling the loop" technique described in Chapter 6 of Mastering Regular Expressions.

remove_comments.py:

import re
import sys

def remove_comments(text):
    """ remove c-style comments.
        text: blob of text with comments (can include newlines)
        returns: text with comments removed
    """
    pattern = r"""
                            ##  --------- COMMENT ---------
           /\*              ##  Start of /* ... */ comment
           [^*]*\*+         ##  Non-* followed by 1-or-more *'s
           (                ##
             [^/*][^*]*\*+  ##
           )*               ##  0-or-more things which don't start with /
                            ##    but do end with '*'
           /                ##  End of /* ... */ comment
         |                  ##  -OR-  various things which aren't comments:
           (                ## 
                            ##  ------ " ... " STRING ------
             "              ##  Start of " ... " string
             (              ##
               \\.          ##  Escaped char
             |              ##  -OR-
               [^"\\]       ##  Non "\ characters
             )*             ##
             "              ##  End of " ... " string
           |                ##  -OR-
                            ##
                            ##  ------ ' ... ' STRING ------
             '              ##  Start of ' ... ' string
             (              ##
               \\.          ##  Escaped char
             |              ##  -OR-
               [^'\\]       ##  Non '\ characters
             )*             ##
             '              ##  End of ' ... ' string
           |                ##  -OR-
                            ##
                            ##  ------ ANYTHING ELSE -------
             .              ##  Anything other char
             [^/"'\\]*      ##  Chars which doesn't start a comment, string
           )                ##    or escape
    """
    regex = re.compile(pattern, re.VERBOSE|re.MULTILINE|re.DOTALL)
    noncomments = [m.group(2) for m in regex.finditer(text) if m.group(2)]

    return "".join(noncomments)

if __name__ == '__main__':
    filename = sys.argv[1]
    code_w_comments = open(filename).read()
    code_wo_comments = remove_comments(code_w_comments)
    fh = open(filename+".nocomments", "w")
    fh.write(code_wo_comments)
    fh.close()

Example:
To test the script, I created a test file called testfile.c:
/* This is a C-style comment. */
This is not a comment.
/* This is another
 * C-style comment.
 */
"This is /* also not a comment */"

Run the script:
To use the script, I put the script, remove_comments.py, and my test file, testfile.c, in the same directory and ran the following command:
python remove_comments.py testfile.c

Results:
The script created a new file called testfile.c.nocomments:
This is not a comment.

"This is /* also not a comment */"



---------------
Minor note on Perl to Python migration:
I modified the original regular expression comments a little bit. In particular, I had to put at least one character after the ## Non "\ and ## Non '\ lines because, in Python, the backslash was escaping the following newline character and the closing parenthesis on the following line was being treated as a comment by the regular expression engine. This is the error I got, before the fix:
$ python remove_comments.py
Traceback (most recent call last):
  File "remove_comments.py", line 39, in <module>
    regex = re.compile(pattern, re.VERBOSE|re.MULTILINE|re.DOTALL)
  File "C:\Programs\Python25\lib\re.py", line 180, in compile
    return _compile(pattern, flags)
  File "C:\Programs\Python25\lib\re.py", line 233, in _compile
    raise error, v # invalid expression
sre_constants.error: unbalanced parenthesis

Blogger expandable post summaries

Here is the link I used to get expandable post summaries on Blogger.

http://hackosphere.blogspot.com/2006/11/selective-expandable-posts.html

This hack avoids showing the "Read more..." link even if there is
nothing more to read. (The Blogger help article had this problem.)
However, this hack still has this problem if you click on the "Newer
Posts" or "Older Posts" at the bottom of the page.

Here is what's put in the post template:
Type your summary here
<span id="fullpost">
Type rest of the post here
</span>

Migrating Excel data to SQLite using Python

In a previous post, I described how I designed a SQLite relational database from an Excel table. It was a small example, so I hardcoded the data into the Python script. For my actual problem, I need to convert my Excel data into a SQLite database automatically. To do this, I used the win32com module and the sqlite3 module included in Python 2.5.

Here is the table from my previous post. It shows some variables in in my C program. It shows the variable name, type, the module it belongs to, and a short description. Here is the table from my previous post. It shows some variables in in my C program. It shows the variable name, type, the module it belongs to, and a short description.

idnamemoduletypedesc
1fooModuleExtdoubleDescription of foo
2barModuleExtdoubleDescription of bar
3knarkModule1intDescription of knark
4wertModule1doubleDescription of wert
5jibModule1doubleDescription of jib
6lazModule2doubleDescription of laz
7kewModule2doubleDescription of kew

After installing the win32com module from http://sourceforge.net/project/platformdownload.php?group_id=78018, I used the following code.

import os
import sqlite3
from win32com.client import constants, Dispatch

#----------------------------------------
# get data from excel file
#----------------------------------------
XLS_FILE = os.getcwd() + "\\example.xls"
ROW_SPAN = (14, 21)
COL_SPAN = (2, 7)
app = Dispatch("Excel.Application")
app.Visible = True
ws = app.Workbooks.Open(XLS_FILE).Sheets(1)
exceldata = [[ws.Cells(row, col).Value 
              for col in xrange(COL_SPAN[0], COL_SPAN[1])] 
             for row in xrange(ROW_SPAN[0], ROW_SPAN[1])]

#----------------------------------------
# create SQL table and fill it with data
#----------------------------------------
conn = sqlite3.connect('example.db')
c = conn.cursor()
c.execute('''CREATE TABLE exceltable (
   id INTEGER,
   name TEXT,
   module TEXT,
   type TEXT,
   desc TEXT
)''')
for row in exceldata:
    c.execute('INSERT INTO exceltable VALUES (?,?,?,?,?)', row)
conn.commit()

#----------------------------------------
# display SQL data
#----------------------------------------
c.execute('SELECT * FROM exceltable')
for row in c:
    print row

The Excel filename is example.xls and the table data begins at B14 (2nd column, 14th row) and ends at F20 (6th column, 20th row) in Sheet 1. The script uses a declarative approach to store the data in a Python list of lists. It creates a SQLite database named example.db and creates a connection to it. It then fills the database using the Python list data structure. Finally, it displays the newly created database. The screen output is shown below.

(1, u'foo', u'ModuleExt', u'double', u'Description of foo')
(2, u'bar', u'ModuleExt', u'double', u'Description of bar')
(3, u'knark', u'Module1', u'int', u'Description of knark')
(4, u'wert', u'Module1', u'double', u'Description of wert')
(5, u'jib', u'Module1', u'double', u'Description of jib')
(6, u'laz', u'Module2', u'double', u'Description of laz')
(7, u'kew', u'Module2', u'double', u'Description of kew')

If I want to interact with the newly created database using SQLite directly, I can run sqlite3 example.db from my Cygwin bash command line. (Note the conn.commit() line in my Python script is very important for this step to work. For some reason, I didn't see this in the Python documentation.) Here is an example session using SQLite directly with the new database.

$ sqlite3 example.db
SQLite version 3.5.1
Enter ".help" for instructions
sqlite> .schema
CREATE TABLE exceltable (
   id INTEGER,
   name TEXT,
   module TEXT,
   type TEXT,
   desc TEXT
);
sqlite> select * from exceltable;
1|foo|ModuleExt|double|Description of foo
2|bar|ModuleExt|double|Description of bar
3|knark|Module1|int|Description of knark
4|wert|Module1|double|Description of wert
5|jib|Module1|double|Description of jib
6|laz|Module2|double|Description of laz
7|kew|Module2|double|Description of kew
sqlite>

Cygwin install tips using cyg-apt

I recently got a new computer at work and needed to reinstall Cygwin. Here is my install procedure using the very nice cyg-apt script. It's like Debian's apt-get for Cygwin! Note, this install process assumes that I have all my init files backed up and can copy them over to my new machine.

  1. Run the Cygwin setup program and install the default packages plus wget and python.
  2. Set the Windows user environment variable HOME to c:\home\sofeng. Note, I tried it with forward slashes, but then it doesn't work in my Windows batch file.
  3. Start the Cygwin bash shell
  4. Change the cygdrive prefix:
    mount -s --change-cygdrive-prefix /
    If you don't have admin privileges:
    mount -u --change-cygdrive-prefix /
  5. Copy over my init files and startup scripts:
    • ~/bin/startxwin.bat
      This is the Windows batch file that I use to start my Cygwin/X environment. It does the following:
      • Adds ~/bin to my PATH
      • cds to my HOME directory
      • Starts the ratpoison window manager with a urxvt terminal running screen
    • ~/.Xdefaults
      Sets the colors, font, and scrolling behavior for urxvt.
    • ~/.bashrc
      • Sets the PS1 (prompt), EDITOR, http_proxy, and other environment variables.
      • Sets a lot of aliases
      • Sets bash filename completion to be case insensitive
    • ~/.cyg-apt
    • ~/.emacs
    • ~/.emacs.d
    • ~/.inputrc
    • ~/.screenrc
    • /etc/ssmtp/ssmtp.conf
    • /var/cron/tabs/sofeng
  6. Install cyg-apt
    $ wget http://www.lilypond.org/~janneke/software/cyg-apt
    $ chmod a+xr cyg-apt
    $ mv cyg-apt ~/bin
  7. Install good stuff (note, if you don't already have ~/.cyg-apt, you will need to run cyg-apt setup first):
    $ cyg-apt install rsyncBest backup utility
    $ cyg-apt install screenTerminal multiplexer
    $ cyg-apt install xorg-x11-baseX Windows
    $ cyg-apt install rxvt-unicode-XBetter Xterm
    $ cyg-apt install diffutilsdiff
    $ cyg-apt install gcc-coreGCC C complier
    $ cyg-apt install makeGNU make
    $ cyg-apt install xorg-x11-develX header and library files for compiling ratpoison or other programs
    $ cyg-apt install readlineused for ratpoison
    $ cyg-apt install cronso I can schedule my backups and updatedb
    $ cyg-apt install ssmtpSendmail replacement so cron can email me
    $ cyg-apt install emacsTerminal version of emacs for quick editing jobs from the shell

Other notes:

  • I thought that I needed to add c:/cygwin/bin to my Windows path in order to use grep in Emacs Windows. However, I later found out that this step is not needed. It is generally bad practice to add c:/cygwin/bin to the Windows path. Instead, I added the required code in my .emacs file. See http://cygwin.com/faq/faq-nochunks.html#faq.using.ntemacs for more information.
  • I used to keep my http proxy server information in ~/.wgetrc, but now I store it in the http_proxy environment variable. (Note, it is lowercase and not uppercase. I don't know why this is.)