Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

    

Saving a Python dict to a file using pickle

Per Programming Python, 3rd Edition, there are a number of methods to store persistent data with Python:

  • I often use flat files to read or write text (string) data using the os library.
  • Flat files are read sequentially, but dbm files allow for keyed access to string data
  • The pickle module can be used to store non-string Python data structures, such as Python dicts. However, the data is not keyed as with dbm files.
  • shelve files combine the best of the dbm and pickle methods by storing pickled objects in dbm keyed files.
  • I've read good things about the ZODB object-oriented database , but I don't know too much about it. Per the book, it is a more powerful alternative to shelves.
  • The final option is interfacing with a full-fledged SQL relational databases. As I mentioned before, Python 2.5 has an interface to SQLite as part of the standard distribution.

Here is an example using pickle which writes a Python dict to a file and reads it back again:

import pickle

# write python dict to a file
mydict = {'a': 1, 'b': 2, 'c': 3}
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict2 = pickle.load(pkl_file)
pkl_file.close()

print mydict
print mydict2

Results:
{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2}

14 Comments — feed icon Comments feed for this post


#1 code43 commented on 2009-11-18:

hi Eliot, here's an appropriate Python module for you:

http://yserial.sourceforge.net

The module is instructive in the way it unifies the standard batteries: sqlite3 (as of Python v2.5), zlib (for compression), and cPickle (for securely serializing objects).

If your Python program requires data persistance, then y_serial is a module which should be worth importing. Objects are warehoused in a database file in very compressed form. Steps for insertion, organization by annotation, and finally retrieval are amazingly simple.

Hope this helps...


#2 Eliot commented on 2009-11-18:

code43, looks good, thanks!


#3 __django__ commented on 2010-05-26:

Thanks for code. I looking for.


#4 PATX commented on 2010-06-29:

I always refer back to this post every time I do something with dicts... Well most of the time :P. I'm forgetful and this is super helpful.


#5 james commented on 2010-09-16:

Hello ,

How to append dictionary values to a file?

I mean how to add dictionary values to a file without overwritting the existing dictionary values


#6 Chrisfs commented on 2010-11-06:

James, If you want to append data to a text file, you want to open it with the option 'a'. So the code would look like newfile = open('myfile.txt',a)

Then when you wrote the data back, it would append it. I don't know how well that would work with a pickled file.


#7 Eliot commented on 2010-12-09:

James:

Maybe you want something like this?

import pickle

# write python dict to a file
mydict = {'a': 1, 'b': 2, 'c': 3}
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()
print mydict

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict = pickle.load(pkl_file)
pkl_file.close()
print mydict

# update dict and write to the file again
mydict.update({'d': 4})
output = open('myfile.pkl', 'wb')
pickle.dump(mydict, output)
output.close()

# read python dict back from the file
pkl_file = open('myfile.pkl', 'rb')
mydict = pickle.load(pkl_file)
pkl_file.close()
print mydict

Results:

{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2}
{'a': 1, 'c': 3, 'b': 2, 'd': 4}

#8 Jose commented on 2011-04-04:

How to load dictionaries from a pickled file that contains many dictionaries. So far I have been able to load only the first dictionary, and I don't know how to iterate file to load every dictionary and print it.Please help :)


#9 raja commented on 2011-06-24:

Hii,what if wants to replace the value of a particular key of the dictionary,i messed with this.


#10 gudGuy commented on 2011-07-25:

@Jose

well may be i don't know the exact method to save and retrieve multiple dictionaries in pickle, but one thing you can do is that you can make dictionary of dictionaries.

e.g d = {'one': {'a': 1, 'b': 2}, 'two': {'c': 3, 'e': 4}}


#11 Paul commented on 2012-01-18:

@Jose

You can just use pickle.load repeatedly until you get an EOFError exception. Each successive call will give you the next pickled object in the file.


#12 Charl commented on 2012-04-11:

Thanks guy, this works great for loading dictionaries as fixtures for unit tests.


#13 Eliot commented on 2012-04-22:

Charl, I'm glad that worked good for you. Another option for saving fixture data for unit tests is saving the data as json in a file with e.g. the json module in the standard library: http://docs.python.org/library/json.html (I know that method wasn't in the list above. :)


#14 Nitrogen commented on 2012-07-04:

Dear Elliot, Thanks a lot for this post. I have been working on this for days and still not able to figure it out. However I have a similar Challenge to what Jose has posted above. I am trying to save these details(id, name, amount)about each customer in the flat file and then read a particular record if say it's id is typed in. so for example i receive the customer id as a raw_input and and display his name and amount.

Post a comment

Required
Required, but not displayed
Optional

Format using Markdown. (No HTML.)
  • Code blocks: prefix each line by at least 4 spaces or 1 tab (and a blank line before and after)
  • Code span: surround with backticks
  • Blockquotes: prefix lines to be quoted with >
  • Links: <URL>
  • Links w/ description: [description](URL)
Created with Django | Hosted by Linode