Every programming language has strengths and weaknesses. Python offers many convenient programming conventions, but it is computationally slow. Rust gives you machine-level speed and great memory security, but it’s more complex than Python. The good news is that you can mix the two languages, taking advantage of the ease of use of Python to take advantage of the speed and power of Rust. The PyO3 project allows you to get the best of both worlds by writing Python extensions to Rust.
With PyO3, you write Rust code, indicate how it interacts with Python, then compile Rust and deploy it directly to a virtual Python environment, where you can use it unobtrusively with your Python code.
This article is a quick tour of how PyO3 works. You will learn how to set up a Python project with a PyO3 createhow to expose Rust functions as a Python module and how to create Python objects as classes and exceptions in Rust.
Setting up a Python project with PyO3
To start creating a PyO3 project, you need to start with a virtual Python environment, or venv. This is not only to keep your Python project organized, but also to provide a place to install the Rust box that you will build with PyO3. (If you haven’t already installed the Rust toolchain, do so now.)
The exact organization of project directories can vary. In the examples shown in the PyO3 documentation, the PyO3 project is embedded in a directory that contains the Python project and its virtual environment. Another approach is to create two subdirectories: one for your Python project and your venv, and the other for the PyO3 project. The latter approach makes it easier to keep things organized, so we’ll:
- Create a new directory to house your Python and Rust projects. we will call them
pyexampleandrustexamplerespectively. - In it
pyexampledirectory, create your virtual environment and activate it. We’ll eventually add some python code here. It’s important that you do all of your work with Rust and Python code in your venv. - On your activated venv, install the
maturinpackage withpip install maturin.maturinis the tool we use to build our Rust project and integrate it with our Python project. - Change to the Rust project directory and type
maturin init. When asked which fasteners to select, choosepyo3. maturinit will then generate a Rust project in that directory, complete with aCargo.tomlfile describing the project. Note that the project will be given the same name as the directory it is in; in this case it will berustexample.
Rust works in a PyO3 project
When you scaffold a PyO3 project with maturinautomatically creates a code stub file in src/lib.rs. This stub contains code for two functions: a single sample function, sum_as_stringand a function named after your project that exposes other functions as a Python module.
here is an example sum_as_string function:
#[pyfunction]
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
Ok((a + b).to_string())
}
He #[pyfunction] macro, of the pyo3 crate indicates that a given function should be wrapped with an interface to Python. The arguments it takes and the results it returns are translated to and from Python types automatically. (It’s also possible to specify native Python types to receive and return; more on that later.)
In this example, sum_as_string takes two arguments that must be translatable to a native Rust 64-bit integer. For such a case, a Python program would pass in two Python int types. But even then, you would have to be careful: those int types would have to be expressible as a 64 bit integer. if you passed 2**65 to this function, you would get a runtime error because such a large number cannot be expressed as a 64-bit integer. (We’ll talk about another way around this limitation later.)
The return value of this function is a native Python type: a PyResult object that contains a String. The last line of the function returns a Stringwhich the PyO3 wrapper automatically wraps as a Python object.
It is also possible for pyfunction to describe the signature that a given function will accept, for example if you want to accept multiple positional or keyword arguments.
Python and Rust types in PyO3 functions
You’ll want to become familiar with how Python and Rust types map to each other, and make some decisions about which types to use.
Your function can accept Rust types that are automatically converted from Python types, but this means that containers such as dictionaries must be fully converted at the function boundary. That could be slow if you pass a large object, like a list with thousands of objects. To that end, this is best done if you’re passing a single value, such as an integer or float, or container objects that you know aren’t going to have many elements.
You can also accept native Python types at the function boundary and use native Python methods to access them inside the function. This is faster at the function limit, so it’s a better choice if you’re passing container objects with an indeterminate number of elements. But accessing container objects requires the use of native Python methods that are bound by the GIL (Global Interpreter Lock), so you’ll need to cast any object values to native Rust types for speed.
Python modules in a PyO3 project
pyfunction functions themselves are not directly exposed to Python through a module. To do this, we need to create a Python module object via PyO3 and expose our pyfunction works through it.
He lib.rs The archive already has a basic version created for you, which looks like this:
#[pymodule]
fn rustexample(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_function(wrap_pyfunction!(sum_as_string, m)?)?;
Ok(())
}
He pymodule macro indicates that the function in question will be exposed as a module to Python, with the same name (rustexample). We take each of the previously defined functions and expose them throughout the module using the .add_function method. This may seem a bit repetitive, but it provides flexibility when building the module, for example by allowing you to create sub-modules if needed.
Compiling a PyO3 project
Usually, compiling your PyO3 project for use in Python is pretty simple:
- If you have not already done so, activate the virtual environment where you installed
maturin. - Set your Rust project as your current working directory.
- run the command
maturin devto build your project.
The results should be something like this:
(.env) PS D:\Dev\pyo3-article\rustexample> maturin dev -r
Updating crates.io index
[ ... snip ... ]
Downloaded 10 crates (3.2 MB) in 2.50s (largest was `windows-sys` at 2.6 MB)
🔗 Found pyo3 bindings
🐍 Found CPython 3.11 at D:\Dev\pyo3-article\pyexample\.env\Scripts\python.exe
[ ... snip ... ]
Compiling rustexample v0.1.0 (D:\Dev\pyo3-article\rustexample)
Finished release [optimized] target(s) in 10.86s
📦 Built wheel for CPython 3.11 to [ ... snip ...]
\.tmpUbXtlF\rustexample-0.1.0-cp311-none-win_amd64.whl
🛠 Installed rustexample-0.1.0
Default, maturin build Rust code in pre-release mode. In this example, we pass the -r flag to maturin to build Rust in release mode.
The resulting code should be installed directly into your virtual environment and you should be able to view it with pip list:
(.env) PS D:\Dev\pyo3-article\rustexample> pip list
Package Version
----------- -------
maturin 0.14.12
pip 23.0
rustexample 0.1.0
setuptools 67.1.0
To test your created package, start the python instance in your virtual environment and try to import the package:
Python 3.11.1 (tags/v3.11.1:a7a450f, Dec 6 2022, 19:58:39)
[MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import rustexample
>>> rustexample
<module 'rustexample' from 'D:\\Dev\\pyo3-article\\pyexample\\
.env\\Lib\\site-packages\\rustexample\\__init__.py'>
It should import and run like any other Python package.
advanced PyO3
So far, you’ve only seen the basics of what PyO3 can do. But PyO3 supports many other Python features, many of which you’ll probably want to interact with in your Rust code.
Large integer support
Python automatically converts integers to “large integers” or integers of arbitrary size. If you want to pass a Python integer object to a PyO3 function and use it like a native Rust big integer, you can do so with pyo3::num_bigint , which uses the existing num_bigint box. Just remember that large integers may not support all operations.
Parallelism
As with Cython, any pure Rust code that doesn’t touch the Python runtime can be executed outside of the Python GIL. You can wrap such a function in the Python::allow_threads method to suspend the GIL while it is running. Again, this has to be pure Rust code with No Python objects in use.
Holding the GIL with Rust lives
PyO3 provides a way to maintain the GIL through Rust’s lifetime mechanism, which gives you a way to have mutable or shared access to Python objects. Different types of objects have different GIL rules.
You can access a generic Python object with the PyAny type, or you can use more precise types like PyTuple either PyList. These are a bit faster, since PyO3 can generate code specific to those types. Regardless of the types you use, you should assume that you need to maintain the GIL for as long as you are working with the object.
If you want a reference to a Python object outside of the GIL, for example if you’re storing a Python object reference in a Rust structure, you can use the Py<T> either PyObject (essentially Py<PyAny>) guys.
For a Rust object wrapped in a Python object (containing GIL), yes, this is possible! You can use PyCell<T>. Typically you would do this if you wanted to access the Rust object while maintaining its Rust reference rules and aliases. In that case, the behavior of the Python wrapper object doesn’t interfere with what you want to do. In the same way, you can use PyRef<T> and PyRefMut<T> to obtain static and mutable borrowed references to such objects.
Classes
You can define Python classes in PyO3 modules. If you add the #[pyclass] attribute to a Rust structure or an enum without a field, can be treated as the basic data structure for a class. To add instance methods you would use #[pymethods] with a impl block for the class that contains the functions to use as methods. It is also possible to create class methods, attributes, magic methods, spaces, callable classes, and many other common behaviors.
Note that Rust behaviors impose some limitations. You cannot provide lifetime parameters for classes; everyone has to work like 'static. You also can’t use generic parameters on types that are used as Python classes.
exceptions
Python exceptions in PyO3 can be created in Rust code with the create_exception! macro, or by importing one of the few predefined standard exceptions with the import_exception! macro. Note that, as with functions, you must manually add exceptions created by PyO3 to a module to make them available to Python.
Conclusion
For a long time, creating Python extensions generally meant learning C with all its minimalism and lack of native security. Or, you could use a tool like Cython with all its idiosyncrasies. But for developers who already know Rust and want to use it alongside Python, PyO3 provides a convenient and powerful way to do it.
Copyright © 2023 IDG Communications, Inc.
Be First to Comment