Listing Files In A Directory With Python When The Directory Is Huge

I'm trying to deal with many files in Python. I first need to get a list of all the files in a single directory. At the moment, I'm using: os.listdir(dir) However. This isn't feas

Solution 1:

You may want to try using the scandir module:

scandir is a module which provides a generator version of os.listdir() that also exposes the extra file information the operating system returns when you iterate a directory. scandir also provides a much faster version of os.walk(), because it can use the extra file information exposed by the scandir() function.

There's an accepted PEP proposing to merge it into the Python standard library, so it seems to have some traction.

Simple usage example from their docs:

defsubdirs(path):
    """Yield directory names not starting with '.' under given path."""for entry in os.scandir(path):
        ifnot entry.name.startswith('.') and entry.is_dir():
            yield entry.name

Solution 2:

You could use glob.iglob to avoid reading the entire list of filenames into memory. This returns a generator object allowing you to step through the filenames in your directory one by one:

import glob

files = glob.iglob(pathname\*)

for f in files:
    # do something with f