Python: Multiprocessing.Pipe And Redirecting Stdout

I am using multiprocessing package to spawn a second process from which I would like to redirect stdout and stderr into the first process. I am using multiprocessing.Pipe object: d

Solution 1:

After experimenting in Python 2.7 I got this working example. With os.dup2 pipe's file descriptor is copied to standard output file descriptor, and each print function ends up writing to a pipe.

import os
import multiprocessing


def tester_method(w):
    os.dup2(w.fileno(), 1)

    for i in range(3):
        print 'This is a message!'


if __name__ == '__main__':
    r, w = multiprocessing.Pipe()

    reader = os.fdopen(r.fileno(), 'r')

    process = multiprocessing.Process(None, tester_method, 'TESTER', (w,))
    process.start()

    for i in range(3):
        print 'From pipe: %s' % reader.readline()

    reader.close()
    process.join()

Output:

From pipe: This is a message!

From pipe: This is a message!

From pipe: This is a message!

Solution 2:

The existing answer works for the raw file descriptors, but this may be useful for using Pipe.send() and recv:

    class PipeTee(object):
        def __init__(self, pipe):
            self.pipe = pipe
            self.stdout = sys.stdout
            sys.stdout = self

        def write(self, data):
            self.stdout.write(data)
            self.pipe.send(data)

        def flush(self):
            self.stdout.flush()

        def __del__(self):
            sys.stdout = self.stdout

To use this, create the object in your multiprocess function, pass it the write side of multiprocessing.Pipe, and then use the read side on the parent process with recv, using poll to check if data exists.