Python Runtime: Recompiling And Reusing C Library

I am developing a tool for some numerical analysis of user-defined functions. The idea is to make a convenient UI in Python, where user can enter C function, then press a button -

Solution 1:

Python has really good multiprocessing support (and really not very good threading support), so you could spawn a new python process for each expression to be evaluated, compile and load the dll in the new process, then have it communicate the results back to the parent process. When the spawned process exits, everything should be unloaded.

Solution 2:

Each time the user enters a function, you could generate a new module with a random, unique name (as you might do for a temp file). Then compile that module, load it, and call the function.

Over time, this will of course result in a lot of modules being loaded. If users don't run too many functions in a single session, you might get away with it. Alternatively, you could compile the function within a plain DLL instead of a Python module, and load it with ctypes. When you're done with the DLL, simply discard any references to it -- when the ctypes handle is eventually gced, it should unload the DLL (see How can I unload a DLL using ctypes in Python?).

You will also probably want to delete your DLLs from disk. If you're using Windows, that might be tricky, since it's hard to predict when ctypes will unload it, and Windows won't let you delete it will it's in use.

Solution 3:

Have you considered Weave or Instant (less up-to-date)? I've only used instant myself, but it seems like either would be perfect for what you're describing.

They can both automatically compile and cache c code at runtime from a string. I've only ever used instant to inline c, but I'm sure it or weave would work just as well for compiling user input.

Here's an example of how to use weave, from the tutorial.

>>> a = 'string'
>>> def protected_printf(a):
...     assert(type(a) == type(1))
...     weave.inline(r'printf("%d\n",a);',['a'])
>>> protected_printf(1)
 1
>>> protected_printf('string')
AssertError...