faq.rst 14 KB

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271272273274275276277278279280281282283284285286287288289290291292293294295296297298299300301302303304305306307308309310311312313314315316317318319320321322323324325326327328329330331332333334335336337338339340341342343
  1. Frequently asked questions
  2. ##########################
  3. "ImportError: dynamic module does not define init function"
  4. ===========================================================
  5. 1. Make sure that the name specified in PYBIND11_MODULE is identical to the
  6. filename of the extension library (without suffixes such as .so)
  7. 2. If the above did not fix the issue, you are likely using an incompatible
  8. version of Python (for instance, the extension library was compiled against
  9. Python 2, while the interpreter is running on top of some version of Python
  10. 3, or vice versa).
  11. "Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
  12. ========================================================================
  13. See the first answer.
  14. "SystemError: dynamic module not initialized properly"
  15. ======================================================
  16. See the first answer.
  17. The Python interpreter immediately crashes when importing my module
  18. ===================================================================
  19. See the first answer.
  20. .. _faq_reference_arguments:
  21. Limitations involving reference arguments
  22. =========================================
  23. In C++, it's fairly common to pass arguments using mutable references or
  24. mutable pointers, which allows both read and write access to the value
  25. supplied by the caller. This is sometimes done for efficiency reasons, or to
  26. realize functions that have multiple return values. Here are two very basic
  27. examples:
  28. .. code-block:: cpp
  29. void increment(int &i) { i++; }
  30. void increment_ptr(int *i) { (*i)++; }
  31. In Python, all arguments are passed by reference, so there is no general
  32. issue in binding such code from Python.
  33. However, certain basic Python types (like ``str``, ``int``, ``bool``,
  34. ``float``, etc.) are **immutable**. This means that the following attempt
  35. to port the function to Python doesn't have the same effect on the value
  36. provided by the caller -- in fact, it does nothing at all.
  37. .. code-block:: python
  38. def increment(i):
  39. i += 1 # nope..
  40. pybind11 is also affected by such language-level conventions, which means that
  41. binding ``increment`` or ``increment_ptr`` will also create Python functions
  42. that don't modify their arguments.
  43. Although inconvenient, one workaround is to encapsulate the immutable types in
  44. a custom type that does allow modifications.
  45. An other alternative involves binding a small wrapper lambda function that
  46. returns a tuple with all output arguments (see the remainder of the
  47. documentation for examples on binding lambda functions). An example:
  48. .. code-block:: cpp
  49. int foo(int &i) { i++; return 123; }
  50. and the binding code
  51. .. code-block:: cpp
  52. m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
  53. How can I reduce the build time?
  54. ================================
  55. It's good practice to split binding code over multiple files, as in the
  56. following example:
  57. :file:`example.cpp`:
  58. .. code-block:: cpp
  59. void init_ex1(py::module_ &);
  60. void init_ex2(py::module_ &);
  61. /* ... */
  62. PYBIND11_MODULE(example, m) {
  63. init_ex1(m);
  64. init_ex2(m);
  65. /* ... */
  66. }
  67. :file:`ex1.cpp`:
  68. .. code-block:: cpp
  69. void init_ex1(py::module_ &m) {
  70. m.def("add", [](int a, int b) { return a + b; });
  71. }
  72. :file:`ex2.cpp`:
  73. .. code-block:: cpp
  74. void init_ex2(py::module_ &m) {
  75. m.def("sub", [](int a, int b) { return a - b; });
  76. }
  77. :command:`python`:
  78. .. code-block:: pycon
  79. >>> import example
  80. >>> example.add(1, 2)
  81. 3
  82. >>> example.sub(1, 1)
  83. 0
  84. As shown above, the various ``init_ex`` functions should be contained in
  85. separate files that can be compiled independently from one another, and then
  86. linked together into the same final shared object. Following this approach
  87. will:
  88. 1. reduce memory requirements per compilation unit.
  89. 2. enable parallel builds (if desired).
  90. 3. allow for faster incremental builds. For instance, when a single class
  91. definition is changed, only a subset of the binding code will generally need
  92. to be recompiled.
  93. "recursive template instantiation exceeded maximum depth of 256"
  94. ================================================================
  95. If you receive an error about excessive recursive template evaluation, try
  96. specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
  97. culprit is generally the generation of function signatures at compile time
  98. using C++14 template metaprogramming.
  99. .. _`faq:hidden_visibility`:
  100. "‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]"
  101. ============================================================================================================
  102. This error typically indicates that you are compiling without the required
  103. ``-fvisibility`` flag. pybind11 code internally forces hidden visibility on
  104. all internal code, but if non-hidden (and thus *exported*) code attempts to
  105. include a pybind type (for example, ``py::object`` or ``py::list``) you can run
  106. into this warning.
  107. To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
  108. compiling pybind code.
  109. As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
  110. have been compiled under different versions of pybind itself, it is also
  111. important that the symbols defined in one module do not clash with the
  112. potentially-incompatible symbols defined in another. While Python extension
  113. modules are usually loaded with localized symbols (under POSIX systems
  114. typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
  115. can be changed, but even if it isn't it is not always enough to guarantee
  116. complete independence of the symbols involved when not using
  117. ``-fvisibility=hidden``.
  118. Additionally, ``-fvisibility=hidden`` can deliver considerably binary size
  119. savings. (See the following section for more details).
  120. .. _`faq:symhidden`:
  121. How can I create smaller binaries?
  122. ==================================
  123. To do its job, pybind11 extensively relies on a programming technique known as
  124. *template metaprogramming*, which is a way of performing computation at compile
  125. time using type information. Template metaprogamming usually instantiates code
  126. involving significant numbers of deeply nested types that are either completely
  127. removed or reduced to just a few instructions during the compiler's optimization
  128. phase. However, due to the nested nature of these types, the resulting symbol
  129. names in the compiled extension library can be extremely long. For instance,
  130. the included test suite contains the following symbol:
  131. .. only:: html
  132. .. code-block:: none
  133. _​_​Z​N​8​p​y​b​i​n​d​1​1​1​2​c​p​p​_​f​u​n​c​t​i​o​n​C​1​I​v​8​E​x​a​m​p​l​e​2​J​R​N​S​t​3​_​_​1​6​v​e​c​t​o​r​I​N​S​3​_​1​2​b​a​s​i​c​_​s​t​r​i​n​g​I​w​N​S​3​_​1​1​c​h​a​r​_​t​r​a​i​t​s​I​w​E​E​N​S​3​_​9​a​l​l​o​c​a​t​o​r​I​w​E​E​E​E​N​S​8​_​I​S​A​_​E​E​E​E​E​J​N​S​_​4​n​a​m​e​E​N​S​_​7​s​i​b​l​i​n​g​E​N​S​_​9​i​s​_​m​e​t​h​o​d​E​A​2​8​_​c​E​E​E​M​T​0​_​F​T​_​D​p​T​1​_​E​D​p​R​K​T​2​_
  134. .. only:: not html
  135. .. code-block:: cpp
  136. __ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
  137. which is the mangled form of the following function type:
  138. .. code-block:: cpp
  139. pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
  140. The memory needed to store just the mangled name of this function (196 bytes)
  141. is larger than the actual piece of code (111 bytes) it represents! On the other
  142. hand, it's silly to even give this function a name -- after all, it's just a
  143. tiny cog in a bigger piece of machinery that is not exposed to the outside
  144. world. So we'll generally only want to export symbols for those functions which
  145. are actually called from the outside.
  146. This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
  147. and Clang, which sets the default symbol visibility to *hidden*, which has a
  148. tremendous impact on the final binary size of the resulting extension library.
  149. (On Visual Studio, symbols are already hidden by default, so nothing needs to
  150. be done there.)
  151. In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
  152. potential serious issues when loading multiple modules and is required for
  153. proper pybind operation. See the previous FAQ entry for more details.
  154. Working with ancient Visual Studio 2008 builds on Windows
  155. =========================================================
  156. The official Windows distributions of Python are compiled using truly
  157. ancient versions of Visual Studio that lack good C++11 support. Some users
  158. implicitly assume that it would be impossible to load a plugin built with
  159. Visual Studio 2015 into a Python distribution that was compiled using Visual
  160. Studio 2008. However, no such issue exists: it's perfectly legitimate to
  161. interface DLLs that are built with different compilers and/or C libraries.
  162. Common gotchas to watch out for involve not ``free()``-ing memory region
  163. that that were ``malloc()``-ed in another shared library, using data
  164. structures with incompatible ABIs, and so on. pybind11 is very careful not
  165. to make these types of mistakes.
  166. How can I properly handle Ctrl-C in long-running functions?
  167. ===========================================================
  168. Ctrl-C is received by the Python interpreter, and holds it until the GIL
  169. is released, so a long-running function won't be interrupted.
  170. To interrupt from inside your function, you can use the ``PyErr_CheckSignals()``
  171. function, that will tell if a signal has been raised on the Python side. This
  172. function merely checks a flag, so its impact is negligible. When a signal has
  173. been received, you must either explicitly interrupt execution by throwing
  174. ``py::error_already_set`` (which will propagate the existing
  175. ``KeyboardInterrupt``), or clear the error (which you usually will not want):
  176. .. code-block:: cpp
  177. PYBIND11_MODULE(example, m)
  178. {
  179. m.def("long running_func", []()
  180. {
  181. for (;;) {
  182. if (PyErr_CheckSignals() != 0)
  183. throw py::error_already_set();
  184. // Long running iteration
  185. }
  186. });
  187. }
  188. CMake doesn't detect the right Python version
  189. =============================================
  190. The CMake-based build system will try to automatically detect the installed
  191. version of Python and link against that. When this fails, or when there are
  192. multiple versions of Python and it finds the wrong one, delete
  193. ``CMakeCache.txt`` and then add ``-DPYTHON_EXECUTABLE=$(which python)`` to your
  194. CMake configure line. (Replace ``$(which python)`` with a path to python if
  195. your prefer.)
  196. You can alternatively try ``-DPYBIND11_FINDPYTHON=ON``, which will activate the
  197. new CMake FindPython support instead of pybind11's custom search. Requires
  198. CMake 3.12+, and 3.15+ or 3.18.2+ are even better. You can set this in your
  199. ``CMakeLists.txt`` before adding or finding pybind11, as well.
  200. Inconsistent detection of Python version in CMake and pybind11
  201. ==============================================================
  202. The functions ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
  203. provided by CMake for Python version detection are modified by pybind11 due to
  204. unreliability and limitations that make them unsuitable for pybind11's needs.
  205. Instead pybind11 provides its own, more reliable Python detection CMake code.
  206. Conflicts can arise, however, when using pybind11 in a project that *also* uses
  207. the CMake Python detection in a system with several Python versions installed.
  208. This difference may cause inconsistencies and errors if *both* mechanisms are
  209. used in the same project. Consider the following CMake code executed in a
  210. system with Python 2.7 and 3.x installed:
  211. .. code-block:: cmake
  212. find_package(PythonInterp)
  213. find_package(PythonLibs)
  214. find_package(pybind11)
  215. It will detect Python 2.7 and pybind11 will pick it as well.
  216. In contrast this code:
  217. .. code-block:: cmake
  218. find_package(pybind11)
  219. find_package(PythonInterp)
  220. find_package(PythonLibs)
  221. will detect Python 3.x for pybind11 and may crash on
  222. ``find_package(PythonLibs)`` afterwards.
  223. There are three possible solutions:
  224. 1. Avoid using ``find_package(PythonInterp)`` and ``find_package(PythonLibs)``
  225. from CMake and rely on pybind11 in detecting Python version. If this is not
  226. possible, the CMake machinery should be called *before* including pybind11.
  227. 2. Set ``PYBIND11_FINDPYTHON`` to ``True`` or use ``find_package(Python
  228. COMPONENTS Interpreter Development)`` on modern CMake (3.12+, 3.15+ better,
  229. 3.18.2+ best). Pybind11 in these cases uses the new CMake FindPython instead
  230. of the old, deprecated search tools, and these modules are much better at
  231. finding the correct Python.
  232. 3. Set ``PYBIND11_NOPYTHON`` to ``TRUE``. Pybind11 will not search for Python.
  233. However, you will have to use the target-based system, and do more setup
  234. yourself, because it does not know about or include things that depend on
  235. Python, like ``pybind11_add_module``. This might be ideal for integrating
  236. into an existing system, like scikit-build's Python helpers.
  237. How to cite this project?
  238. =========================
  239. We suggest the following BibTeX template to cite pybind11 in scientific
  240. discourse:
  241. .. code-block:: bash
  242. @misc{pybind11,
  243. author = {Wenzel Jakob and Jason Rhinelander and Dean Moldovan},
  244. year = {2017},
  245. note = {https://github.com/pybind/pybind11},
  246. title = {pybind11 -- Seamless operability between C++11 and Python}
  247. }