Skip to content Skip to sidebar Skip to footer

Running Multiple Concurrent Python Programs Accessing The Same Database Table

Is there anything in Python that allows you to run multiple concurrent Python programs that could potentially access the same database table and to prevent each program from using

Solution 1:

Several issues:

  1. multiple concurrent Python programs - see http://wiki.python.org/moin/Concurrency, for the start I would try with builtin module multiprocessing (http://docs.python.org/2/library/multiprocessing.html)
  2. access the same database table - each process should create own db connection - after that concurrency is managed by/or configured within rdbms and/or connection/query options. If you really need to sync between processes - using Locks/Semaphores could help.
  3. prevent each program from using the full cpu - it depends what your processes should do, I would go with:
    • having one main program that runs all the time (master process), does pauses (time.sleep, gevent.sleep or similar) and spawns and controls spawned processes (workers)
    • spawned processes do the job (workers) - open new connection, do db actions and quit

I'm sure that some of the workflows/systems provided by multiprocessing (or other) modules could fit your needs (Workers, Pools, Queues, Pipes, Shared states, Synchronization, ...).

Post a Comment for "Running Multiple Concurrent Python Programs Accessing The Same Database Table"