Modify Python Script To Run On Every File In A Directory

so I have a python script which takes the filename as a command argument and processes that file. However, because I have 263 files which need the same processing, I was wondering

Solution 1:

As a practical matter, whatever shell you're using probably has some syntax that can be easily used for this. In Bash, for example:

for f in *; do python myscript.py $f; done

To actually do this in Python, I'd suggest structuring your program so that the main code is in a function which takes one argument, the filename.

defprocess(filename):
    ...code goes here...

Then you can invoke this function like so,

for f in os.listdir(folder):
    process(f)

folder could be passed as a command-line argument, or just written into the script (if it's not something you'd be reusing).

EDIT: In response to your edit, I'd suggest just giving the filenames as regular command-line arguments, without using the -r option, so that they'll wind up in args. Then you can do

for f in args:
    process(f)

or if you would rather pass the directory name as the command-line argument,

for d in args:
    for f in os.listdir(d):
        process(f)

Alternatively, I suppose you could pass multiple instances of the -r option, and then do

for opt, argin opt_list:
    if opt == '-r':
        process(arg)

Solution 2:

When I am working on multiple files/folders, I usually use os.walk:

import os
for root, dirs, files in os.walk(dir):
   for fname in files:
       do_something(fname) 

Get your directory from getopt or optparse. Also, you can build up absolute path with the os.path.abspath if you need it.

current_file = "%s%s%s" % (os.path.abspath(root), os.path.sep, fname)
do_something(current_file)

Solution 3:

os.walk() sounds like it might work here.

deftraverse_and_touch(directory, touch):
  '''
  General function for traversing a local directory. Walks through
  the entire directory, and touches all files with a specified function.
  '''for root, dirs, files in os.walk(directory):
    for filename in files:
      touch(os.path.join(root, filename))
  return

Now, all you need to do is pass in the directory you'd like to traverse and a function and it'll perform the code on every file.

os.walk() also traverses all sub-directories.

Solution 4:

I suggest your 'main' should process each file given after the options. That is, in the "args" variable. Don't pass paths in with "-r ", this limits your flexibility. If you use os.walk() etc in the program you're requiring the system to work only on trees of files, which makes it more difficult to customize and develop.

If the program works with a list of paths, it's very easy to use in different ways. For example, you can list one data file for testing. To process a directory do "myprogram dir/*.dat". To process a tree of files use backquotes:

myprogram `find . -name "*.dat"`

Lastly you can do very cheap parallel processing. Something like:

find . -name '*.dat' | xargs -P 5 myprogram

Five copies of your program are run in parallel. No locking or forks or threads or other synchronization necessary.

(Above assumes you're on a Linux/OSX type system.)

Solution 5:

Yes, you could modify it like that. Loop through the arguments rather than indexing the first element.