Ben Chuanlong Du's Blog

It is never too late to learn.

Speed up Python Using Native Code

Tips and Traps

  1. It is never a good idea to do pre-mature optimization. Always profile (please refer to Profile Performance of Python Applications and Python Profiler for JupyterLab Notebooks) your code before you optimize it. If you do decide to optimize you Python code, there are multiple approaches. First, think about whether you can improve your algorithm. Please refer to Python Performance Tips for some general tips on this. Second, if you are doing numerical computing using numpy arrays, make sure you have leveraged vector/matrix/array computation to speed up your code rather than using slow Python loops. Last but not least, if the 2 above approaches does not help, you think reimplement performance-critical part of your code using faster langages (C, C++, Rust or Java).

Numba

Numba is a numpy aware dynamic Python compiler using LLVM. If numerical computing is the bottle-neck of your code, numba is the firt tool you should try as it is of minimal effort. There is no (or absolute minimal) change to your Python code, you only need to add a few numba decorators. What is great about numba is that with minimal effort, you get equal to (or even better than) C speed.

Please refer to Numba vs. Cython: Take 2 for more details and solid examples on numba.

PyO3 for Rust

PyO3 for Rust is a Rust bindings for the Python interpreter.

Bindings for Java

Please refer to Java Interfaces for Python for more details.

ctypes

  1. ctypes is included in the Python standard library. It is a good choice for C extensions.

  2. Not friendly on C++ code.

cffi

Cython

  1. Cython has wide adoption historically. It gets close to C speed using close to Python implementation. It was a great choice for Python users who are not familiar with C, C++, Rust or Java. However, numba might be a better alternative to Cython now in many situations.

  2. Not friendly on C++ code.

cython3 ma.pyx 
gcc -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python3.5 -o ma.so ma.c

boost.python

boost.python is a C++ library which enables seamless interoperability between C++ and the Python programming language. The library includes support for:

  • References and Pointers
  • Globally Registered Type Coercions
  • Automatic Cross-Module Type Conversions
  • Efficient Function Overloading
  • C++ to Python Exception Translation
  • Default Arguments
  • Keyword Arguments
  • Manipulating Python objects in C++
  • Exporting C++ Iterators as Python Iterators
  • Documentation Strings

In particular, boost.python has integration of numpy C-APIs, which make it the suitable tool if you need to manipulate numpy arrays in C++.

Boost.Python is an enormously large and complex suite of utility libraries that works with almost every C++ compiler in existence. This compatibility has its cost: arcane template tricks and workarounds are necessary to support the oldest and buggiest of compiler specimens.

pybind11

pybind11 is a lightweight header-only library that exposes C++ types in Python and vice versa, mainly to create Python bindings of existing C++ code. Its goals and syntax are similar to the excellent Boost.Python library but is much more light-weighted and user-friendly than boost.python. Due to ease to use and good documentation, pybind11 has a more active community than Boost.Python. In particular, The deep learning framework PyTorch uses pybind11 to interact with C++.

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

#include <pybind11/stl.sh> is for auto type conversion of STL.

g++ -O2 -shared -std=c++11 -I/usr/local/include/python3.5 -I/usr/include/python3.5m -fPIC ma_cpp.cpp -o macpp.so

PYBIND11_PLUGIN(ma_cpp) {

PYBIND11_PLUGIN defines the module name and the compiled shared object must use this name.

The cppimport package makes things easy. Don't forget to use the following setup when using cppimport.

<%
setup_pybind11(cfg)
%>

cppyy

Dynamic,easy to use but sacrifice some performane.

In [ ]:
## [Pythran](https://github.com/serge-sans-paille/pythran)

A claimless Python to c++ converter.

References

http://zhuoqiang.me/bring-cpp-class-to-python-through-cython.html

https://stackoverflow.com/questions/145270/calling-c-c-from-python

http://www.algorithm.co.il/blogs/computer-science/10-python-optimization-tips-and-issues/

如何选择Python与C++之间的胶水

Cython, pybind11, cffi – which tool should you choose?

Use Cython, Numba, or C/C++ for algorithmic code

CPPYY PART III 和pypy/cython/numba的性能对比

In [ ]:

Comments