Posts tagged "files_directories"

A hack to copy files between two remote hosts using Python

Date: 2010-02-08 | Modified: 2010-02-16 | Tags: files_directories, linux, python, ssh, ubuntu | 1 Comment

I sometimes need to copy a file (such as a database dump) between two remote hosts on EC2. Normally this involves a few steps: scp'ing the ssh keyfile to Host 1, ssh'ing to Host 1, looking up the address for Host 2, then scp'ing the desired file from Host 1 to Host 2.

I was excited to read in the man page that scp can copy files between two remote hosts directly. However, it didn't work for me. Apparently, running scp host1:myfile host2: is like running ssh host1 scp myfile host2: so I still need ...

Iterating over lines in multiple Linux log files using Python

Date: 2010-01-23 | Modified: 2010-02-24 | Tags: files_directories, linux, python, ubuntu | 3 Comments

I needed to parse through my Nginx log files to debug a problem. However, the logs are separated into many files, most of them are gzipped, and I wanted the ordering within the files reversed. So I abstracted the logic to handle this into a function. Now I can pass a glob pattern such as /var/log/nginx/cache.log* to my function, and iterate over each line in all the files as if they were one file. Here is my function. Let me know if there is a better way to do this.

Update 2010-02-24:To handle multiple log ...

Notes on working with files and directories in Python

Date: 2009-09-18 | Modified: 2009-11-09 | Tags: files_directories, python

Documentation:

How to list files in a directory

OPTION 1 - os.listdir()
http://docs.python.org/library/os.html#os.listdir

import os
for filename in os.listdir("/usr"):
    print filename

lib
src
bin
local
X11R6
sbin
share
include
lib32
man
games

OPTION 2 - glob.glob()
http://docs.python.org/library/glob.html#glob.glob
```
import glob
for path in glob.glob('/usr ...
```

Working with files and directories in Python

Date: 2008-04-14 | Modified: 2008-10-31 | Tags: files_directories, python

I often have a difficult time finding the appropriate functions to work with files and directories in Python. I think one reason is because the Library Reference puts seemingly related functions in two different places: 11 File and Directory Access and 14.1.4 Files and Directories.

Section 11, File and Directory Access contains documentation for useful functions such as os.path.exists which checks if a path exists, glob.glob which is useful for matching filenames using the Unix-style * and ? wildcards, and shutil.copy which is similary to the Unix cp command.

This section includes a total of 11 ...

How to get current working directory in Python

Date: 2007-11-02 | Tags: files_directories, python

I always forget how to get the current working directory so here it is, mainly for my own reference.

import os 
os.getcwd() 
'C:\\Python25'

backing up with rsync

Date: 2007-08-31 | Modified: 2008-08-01 | Tags: cygwin, files_directories, python, rsync, softwaretools

Here is a python script using rsync to backup my Users directory in Vista to an external hard drive.

import os

cmd = "rsync -avz --exclude '/AppData/'" + \
      "/cygdrive/c/Users/sofeng" + \
      "/cygdrive/f/backup/Users"
os.system(cmd)

Why not to use the backup program that comes with your external hard drive:

Do not, whatever you do, feed your valuable data to a program that is going to save it in a file format that can only be read by that program, or by that kind of computer. Because when the program can’t or the computer can’t, you’re ...

Python os.walk example

Date: 2007-03-15 | Modified: 2008-09-07 | Tags: files_directories, python | 5 Comments

Here is a simple os.walk() example which walks your directory tree and returns the path, a list of directories, and a list of files:

import os

path = "c:\\python25"

i = 0
for (path, dirs, files) in os.walk(path):
    print path
    print dirs
    print files
    print "----"
    i += 1
    if i >= 4:
        break

Here are the results:

c:/python25
['DLLs', 'Doc', 'include', 'Lib', 'libs', 'tcl', 'Tools']
['LICENSE.txt', 'NEWS.txt', 'pylupdate4.exe', 'pyrcc4.exe', 'python.exe', 'pythonw.exe', 'pyuic4.bat', 'README.txt', 'temp.py', 'w9xpopen.exe']
----
c:/python25\DLLs
[]
['bz2.pyd', 'py.ico', 'pyc.ico', 'pyexpat.pyd', 'select.pyd', 'sqlite3 ...

SaltyCrane Blog