Entry

Embedding the Python interpreter in a Windows app

Aug 13th, 2000 05:20
unknown unknown, Edward K. Ream, David Bolen

Embedding the Python interpreter in a Windows app can be summarized as
follows:
1. Do _not_ build Python into your .exe file directly.  On Windows,
Python must be a DLL to handle importing modules that are themselves
DLL's.  (This is the first key undocumented fact.) Instead, link to
python15.dll; it is typically installed in c:\Windows\System.  
You can link to Python statically or dynamically.  Linking statically
means linking against python15.lib The drawback is that your app won't
run if python15.dll does not exist on your system.
General note: python15.lib is the so-called "import lib" corresponding
to python.dll.  It merely defines symbols for the linker.
Borland note: convert python15.lib to OMF format using Coff2Omf.exe
first.
Linking dynamically greatly simplifies link options; everything happens
at run time.  Your code must load python15.dll using the Windows
LoadLibraryEx routine.  The code must also use access routines and data
in python15.dll (that is, Python's C API's) using pointers obtained by
the Windows GetProcAddress routine.  Macros can make using these
pointers transparent to any C code that calls routines in Python's C
API.
2. If you use SWIG, it is easy to create a Python "extension module"
that will make the app's data and methods available to Python.  SWIG
will handle just about all the grungy details for you.  The result is C
code that you link _into your .exe file_ (!)  You do _not_ have to
create a DLL file, and this also simplifies linking.
3.  SWIG will create an init function (a C function) whose name depends
on the name of the extension module.  For example, if the name of the
module is leo, the init function will be called initleo().  If you use
SWIG shadow classes, as you should, the init function will be called
initleoc().  This initializes a mostly hidden helper class used by the
shadow class.
The reason you can link the C code in step 2 into your .exe file is that
calling the initialization function is equivalent to importing the
module into Python! (This is the second key undocumented fact.)
4. In short, you can use the following code to initialize the Python
interpreter with your extension module.
    #include "python.h"
    ...
    Py_Initialize();  // Initialize Python.
    initmyAppc();  // Initialize (import) the helper class. 
    PyRun_SimpleString("import myApp") ;  // Import the shadow class.
5. There are two problems with Python's C API which will become apparent
if you use a compiler other than MSVC, the compiler used to build
python15.dll.
Problem 1: The so-called "Very High Level" functions that take FILE *
arguments will not work in a multi-compiler environment; each compiler's
notion of a struct FILE will be different.  Warnings should be added to
the Python documentation!  From an implementation standpoint these are
very _low_ level functions.
Problem 2: SWIG generates the following code when generating wrappers to
void functions:
    Py_INCREF(Py_None);
    _resultobj = Py_None;
    return _resultobj;
Alas, Py_None is a macro that expands to a reference to a complex data
structure called _Py_NoneStruct inside python15.dll.  Again, this code
will fail in a mult-compiler environment.  Replace such code by:
        return Py_Build("");
It may be possible to use SWIG's %typemap command to make the change
automatically, though I have not been able to get this to work (I'm a
complete SWIG newbie.)
6. Using a Python shell script to put up a Python interpreter window
from inside your Windows app is not a good idea; the resulting window
will be independent of your app's windowing system.  Rather, you (or the
wxPythonWindow class) should create a "native" interpreter window.  It
is easy to connect that window to the Python interpreter.  You can
redirect Python's i/o to _any_ object that supports read and write, so
all you need is a Python object (defined in your extension module) that
contains read and write methods.
------Comments by David Bolen---------
Just a few incomplete thoughts:
* I actually found the Extending and Embedding documentation
  reasonably good - at least enough to let me construct a project I
  needed when I first started with Python - so I'm not sure about the
  undocumented comments.  Yes, embedding is only afforded a small
  amount of space per-se, but aside from interpreter initialization,
  it's almost identical to extending so you can look at the
  documentation in that light.  Of course more info can never hurt, so
  I think your efforts are definitely in the right direction,
  particularly where they highlight issues for cross-compiler
  compatibility.
  To be honest, in my own embedding work, what I missed up front was
  more information about creating objects in C that worked like
  built-in types or classes.  It seems to me you really have to look
  at the Python source itself to get a better handle on that process.
  Then again maybe I'm just wierd - unintentionally, it turned out my
  first use of Python was to embed it within a multi-threaded C
  application, with real-time communication between the Python and
  native C threads.  At the time I definitely ran into some
  uncertainties with respect to the global interpreter thread lock and
  so on, but it was only later that I realized I probably started at
  the wrong end of the curve :-)
* I don't know about anyone else but I find your use of the terms
  "static" and "dynamic" with respect to linking a bit unintuitive.
  To my mind (and when I typically see them used), static linking is
  just that - it statically binds the library code into your
  executable.  So if you were going to statically link Python into
  your executable you would directly bring the Python interpreter in,
  and would have no dependence on python15.dll.  E.g., it would be
  akin to 'freeze'ing Python.  You are right, it would prevent
  dynamically loading other extension modules that depended on the
  DLL, but it's still one possible approach if you really want to have
  no dependency on external DLLs and you know which modules your
  embedded Python scripts need access to.
  On the other hand, dynamic linking is when you link against an
  import library (e.g., python15.lib) in order to reference an
  external DLL at runtime.  You are dynamically linked to the DLL.
  Dynamically "loading" a DLL at run-time is an entirely different
  beast, and isn't a link time "term" at all, since the linking stage
  has no knowledge of the DLL access at all.
  Note also that your comment about "Linking dynamically" (run time
  loading) greatly simplifying link options is certainly true, since
  there aren't any link options involved :-)  But it does make life
  more complicated for the code that has to dynamically load and
  locate the functions to be used.  Even with macros it's a bit of
  work, since you have to deal with potential error conditions that
  can't exist in the other cases (e.g., the inability to even locate
  the function at hand).
  I guess the more recent MS "delay loading" support starts to blur
  the line between these approaches even further (I haven't tried
  delay-loading the Python DLL myself yet).
* While run-time loading would permit your application to run without
  a python DLL on the system, if using the Python interpreter is a key
  part of your application, what's the point?  So in general, unless
  the ability to run without Python is crucial, I'd suggest just going
  with the standard dynamic linking approach as the suggested method.
* On points 2-4, while going the SWIG route is certainly doable, it's
  probably also worth noting that for a simple embedded application
  (in my case I really only offered about 6-12 methods, a bunch of
  fixed constant objects and stdout/stderr objects to reroute the
  Python output to my application) it's not really necessary.  It's
  pretty easy to just define your own function table and write your
  functions, particularly if they aren't pre-existing and the reason
  you are writing them is just to interface to Python.  SWIG is fine
  too, but for simple stuff it's just an extra tool to have to have
  around (and it does create some issues by default, as you noted with
  the Py_None return template it uses).
* You've definitely got a point about the FILE structure - it might be
  interesting to see if there was any way to create a shim module that
  might help out (e.g., a DLL built with MSC to work with with Python
  DLL but exporting the typical C RTL functions that use FILE *).
* For Py_None, should that be Py_BuildValue("") and not Py_Build("")?