Saltycrane logo

SaltyCrane Blog

Notes on Python, Django, and web development on Ubuntu Linux

     Posts tagged "files_directories"

A hack to copy files between two remote hosts using Python

I sometimes need to copy a file (such as a database dump) between two remote hosts on EC2. Normally this involves a few steps: scp'ing the ssh keyfile to Host 1, ssh'ing to Host 1, looking up the address for Host 2, then scp'ing the desired file from Host 1 to Host 2.

I was excited to read in the man page that scp can copy files between two remote hosts directly. However, it didn't work for me. Apparently, running scp host1:myfile host2: is like running ssh host1 scp myfile host2: so I still need ...

... read more »

Iterating over lines in multiple Linux log files using Python

I needed to parse through my Nginx log files to debug a problem. However, the logs are separated into many files, most of them are gzipped, and I wanted the ordering within the files reversed. So I abstracted the logic to handle this into a function. Now I can pass a glob pattern such as /var/log/nginx/cache.log* to my function, and iterate over each line in all the files as if they were one file. Here is my function. Let me know if there is a better way to do this.

Update 2010-02-24:To handle multiple log ...

... read more »

Notes on working with files and directories in Python

Documentation:

How to list files in a directory

... read more »

Working with files and directories in Python

I often have a difficult time finding the appropriate functions to work with files and directories in Python. I think one reason is because the Library Reference puts seemingly related functions in two different places: 11 File and Directory Access and 14.1.4 Files and Directories.

Section 11, File and Directory Access contains documentation for useful functions such as os.path.exists which checks if a path exists, glob.glob which is useful for matching filenames using the Unix-style * and ? wildcards, and shutil.copy which is similary to the Unix cp command.

This section includes a total of 11 ...

... read more »

How to get current working directory in Python

I always forget how to get the current working directory so here it is, mainly for my own reference.
import os 
os.getcwd() 
'C:\\Python25'
... read more »

backing up with rsync

Here is a python script using rsync to backup my Users directory in Vista to an external hard drive.
import os

cmd = "rsync -avz --exclude '/AppData/'" + \
      "/cygdrive/c/Users/sofeng" + \
      "/cygdrive/f/backup/Users"
os.system(cmd)
Why not to use the backup program that comes with your external hard drive:
Do not, whatever you do, feed your valuable data to a program that is going to save it in a file format that can only be read by that program, or by that kind of computer. Because when the program can’t or the computer can’t, you’re ...
... read more »

Python os.walk example

Here is a simple os.walk() example which walks your directory tree and returns the path, a list of directories, and a list of files:
import os

path = "c:\\python25"

i = 0
for (path, dirs, files) in os.walk(path):
    print path
    print dirs
    print files
    print "----"
    i += 1
    if i >= 4:
        break
Here are the results:
c:/python25
['DLLs', 'Doc', 'include', 'Lib', 'libs', 'tcl', 'Tools']
['LICENSE.txt', 'NEWS.txt', 'pylupdate4.exe', 'pyrcc4.exe', 'python.exe', 'pythonw.exe', 'pyuic4.bat', 'README.txt', 'temp.py', 'w9xpopen.exe']
----
c:/python25\DLLs
[]
['bz2.pyd', 'py.ico', 'pyc.ico', 'pyexpat.pyd', 'select.pyd', 'sqlite3 ...
... read more »
Created with Django | Hosted by Slicehost