SaltyCrane Blog — Notes on Javascript and web development

How to remove ^M characters from a file with Python

Use the following Python script to remove ^M (carriage return) characters from your file and replace them with newline characters only. To do this in Emacs, see my notes here.

remove_ctrl_m_chars.py:

import os
import sys
import tempfile


def main():
    filename = sys.argv[1]
    with tempfile.NamedTemporaryFile(delete=False) as fh:
        for line in open(filename):
            line = line.rstrip()
            fh.write(line + '\n')
        os.rename(filename, filename + '.bak')
        os.rename(fh.name, filename)


if __name__ == '__main__':
    main()

Run it

$ python remove_ctrl_m_chars.py myfile.txt 

Documentation

Comments


#1 Harsh commented on :

Why would you need this as a utility when you have dos2unix, whose entire purpose is for this?


#2 Eliot commented on :

Harsh: Because I have a bad case of NIH and because I wasn't familiar with dos2unix. Thanks for the tip; I'll use dos2unix next time.


#3 Kaixi commented on :

Wonderful little script :)


#4 Khalil commented on :

or use vi editor's command

:%s/\r//g


#5 Raena commented on :

This is so great! Such a time saver!. Thank you for posing this.

I ran it from withing a putty session with ./ instead of the $ python


#6 David Cary commented on :

In vim, when I see stray ^M character, I: Save changes. Edit file again, using "dos file format" -- with the "mixed" files I get that have a random mixture of both, this does the Right Thing: it interprets each CRLF as one newline, and also interprets any other LF as one newline. Set the current file format to unix. Save changes (and because the current format is "unix", write only a LF for each newline).

In other words, I type:

:w
:e ++ff=dos
:set ff=unix
:w

There's also a way to script vim so it "cleans" all the source files and other text files and in an entire directory.

"Change end-of-line format for dos-mac-unix"

"How do I convert DOS files to Linux files in vim?"

"Remove carriage return in Unix"