Posts tagged "persistence" « SaltyCrane Blog

Notes on sqlalchemy w/ pyodbc, freetds on Ubuntu

l Date: 2011-09-28 | Modified: 2012-05-02 | Tags: linux, notestoself, persistence, python, sql, ubuntu | 1 Comment

These are my notes for connecting to a MS SQL Server using Python on Linux. My developement environment is Ubuntu 10.10 Maverick. The production environment is Redhat/Centos-based Scientific Linux 6. Here is the layers of stuff:

FreeTDS: an open source ODBC driver. Started with version 0.82 and moved to 0.91. Other options: Microsoft ODBC driver for Linux Free, currently only for 64-bit Redhat. Easysoft ODBC driver Commercial, not free.
unixODBC: driver manager, a thin wrapper around the ODBC driver. Version 2.2 (may move to version 2.3 if we use the Microsoft driver).
pyodbc: python ...

Python MongoDB notes

l Date: 2010-02-22 | Modified: 2010-04-07 | Tags: persistence, python | 6 Comments

MongoDB is a popular new schemaless, document-oriented, NoSQL database. It is useful for logging and real-time analytics. I'm working on a tool to store log files from multiple remote hosts to MongoDB, then analyze it in real-time and print pretty plots. My work in progress is located on github.

Here are my first steps using PyMongo. I store an Apache access log to MongoDB and then query it for the number of requests in the last minute. I am running on Ubuntu Karmic 32-bit (though I think MongoDB really wants to run on 64-bit).

Install and run MongoDB

Download ...

Using Python to write to an Excel / OpenOffice Calc spreadsheet on Ubuntu Linux

l Date: 2010-02-15 | Tags: linux, persistence, python, ubuntu | 11 Comments

Via Matt Harrison's blog post, here is how to write Excel or OpenOffice.org Calc spreadsheet files using Python and the xlwt library. Xlwt is a fork of pyExcelerator which handles only writing spreadsheet files. For reading spreadsheets, see xlrd. Note, these libraries don't use COM, so they will work on non-Windows OSes, such as Linux. For more information, see Matt's blog post. He even has a PDF cheat sheet.

Install pip
Install xlwt
```
sudo pip install xlwt
```

Create an example script:

import xlwt

DATA = (("The Essential Calvin and Hobbes", 1988,),
        ("The Authoritative Calvin and Hobbes", 1990 ...

Saving a Python dict to a file using pickle

l Date: 2008-01-07 | Modified: 2008-09-09 | Tags: persistence, python | 13 Comments

Per Programming Python, 3rd Edition, there are a number of methods to store persistent data with Python:

I often use flat files to read or write text (string) data using the os library.
Flat files are read sequentially, but dbm files allow for keyed access to string data
The pickle module can be used to store non-string Python data structures, such as Python dicts. However, the data is not keyed as with dbm files.
shelve files combine the best of the dbm and pickle methods by storing pickled objects in dbm keyed files.
I've read good things about the ...

Migrating Excel data to SQLite using Python

l Date: 2007-11-27 | Modified: 2008-11-03 | Tags: persistence, python, sql | 6 Comments

In a previous post, I described how I designed a SQLite relational database from an Excel table. It was a small example, so I hardcoded the data into the Python script. For my actual problem, I need to convert my Excel data into a SQLite database automatically. To do this, I used the win32com module and the sqlite3 module included in Python 2.5.

Here is the table from my previous post. It shows some variables in in my C program. It shows the variable name, type, the module it belongs to, and a short description. Here is the table ...

Relational database introduction with Python and SQLite

l Date: 2007-11-14 | Modified: 2008-11-03 | Tags: persistence, python, sql | 2 Comments

I mentioned in a previous post that I have an Excel spreadsheet containing a bunch of information about the development C code I'm working on. It is a large table showing which variables are input and output from which functions. The variable names are in the first column and the function names are in the last several columns of the first row. I use "i" or "o" to denote if a variable is an input to or an output from a particular function. I also have a few columns for the variable type and description. A simplified example is ...