Saving a Python dict to a file using pickle

Date: 2008-01-07 | Modified: 2008-09-09 | Tags: persistence, python | 11 Comments

Per Programming Python, 3rd Edition, there are a number of methods to store persistent data with Python:

I often use flat files to read or write text (string) data using the os library.
Flat files are read sequentially, but dbm files allow for keyed access to string data
The pickle module can be used to store non-string Python data structures, such as Python dicts. However, the data is not keyed as with dbm files.
shelve files combine the best of the dbm and pickle methods by storing pickled objects in dbm keyed files.
I've read good things about the ZODB object-oriented database , but I don't know too much about it. Per the book, it is a more powerful alternative to shelves.
The final option is interfacing with a full-fledged SQL relational databases. As I mentioned before, Python 2.5 has an interface to SQLite as part of the standard distribution.

Here is an example using pickle which writes a Python dict to a file and reads it back again:

import pickle

# write python dict to a file
mydict = {'a': 1, 'b': 2, 'c': 3}
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict2 = pickle.load(pkl_file)
pkl_file.close()

print mydict
print mydict2

Results:

{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2}

Python MongoDB notes — posted 2010-02-22
Using Python to write to an Excel / OpenOffice Calc spreadsheet on Ubuntu Linux — posted 2010-02-15
Migrating Excel data to SQLite using Python — posted 2007-11-27
Relational database introduction with Python and SQLite — posted 2007-11-14

11 Comments — Comments feed for this post

#1 code43 commented on 2009-11-18:

hi Eliot, here's an appropriate Python module for you:

http://yserial.sourceforge.net

The module is instructive in the way it unifies the standard batteries: sqlite3 (as of Python v2.5), zlib (for compression), and cPickle (for securely serializing objects).

If your Python program requires data persistance, then y_serial is a module which should be worth importing. Objects are warehoused in a database file in very compressed form. Steps for insertion, organization by annotation, and finally retrieval are amazingly simple.

Hope this helps...

#2 Eliot commented on 2009-11-18:

code43, looks good, thanks!

#3 __django__ commented on 2010-05-26:

Thanks for code. I looking for.

#4 PATX commented on 2010-06-29:

I always refer back to this post every time I do something with dicts... Well most of the time :P. I'm forgetful and this is super helpful.

#5 james commented on 2010-09-16:

Hello ,

How to append dictionary values to a file?

I mean how to add dictionary values to a file without overwritting the existing dictionary values

#6 Chrisfs commented on 2010-11-06:

James, If you want to append data to a text file, you want to open it with the option 'a'. So the code would look like newfile = open('myfile.txt',a)

Then when you wrote the data back, it would append it. I don't know how well that would work with a pickled file.

#7 Eliot commented on 2010-12-09:

James:

Maybe you want something like this?

import pickle

# write python dict to a file
mydict = {'a': 1, 'b': 2, 'c': 3}
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()
print mydict

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict = pickle.load(pkl_file)
pkl_file.close()
print mydict

# update dict and write to the file again
mydict.update({'d': 4})
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict = pickle.load(pkl_file)
pkl_file.close()
print mydict

Results:

{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2, 'd': 4}

#8 Jose commented on 2011-04-04:

How to load dictionaries from a pickled file that contains many dictionaries. So far I have been able to load only the first dictionary, and I don't know how to iterate file to load every dictionary and print it.Please help :)

#9 raja commented on 2011-06-24:

Hii,what if wants to replace the value of a particular key of the dictionary,i messed with this.

#10 gudGuy commented on 2011-07-25:

@Jose

well may be i don't know the exact method to save and retrieve multiple dictionaries in pickle, but one thing you can do is that you can make dictionary of dictionaries.

e.g d = {'one': {'a': 1, 'b': 2}, 'two': {'c': 3, 'e': 4}}

#11 Paul commented on 2012-01-18:

@Jose

You can just use pickle.load repeatedly until you get an EOFError exception. Each successive call will give you the next pickled object in the file.


Name:	Required
Email:	Required, but not displayed
Website:	Optional
Comment:	Format using Markdown. (No HTML.) Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after) Code span: surround with backticks Blockquotes: prefix lines to be quoted with > Links: <URL> Links w/ description: [description](URL)

SaltyCrane Blog