Interoperability of rust and python

Context : Rust uses programming concepts , which are either implemented differently in python or not implemented altogether. As in, unlike python, rust tells, that no variable is mutable in rust until you specify so, implements borrow checks, macros and lifetimes, to begin with.

This gets memory gurantees, way less boilerplate, among other things ,which the build wouldn’t have had in pure python.

Pyo3 genreates API ¹ for Python types in Rust, and most of the SQL ops, so you almost always write a python client to do that thing, you were using it otherwise , esp for data ops ml, NLP.

Truth

I was always biased against python, esp learning it with frameworks like dJango, but it’s seemless utility made me look at it and I began to work for workarounds , as soon as I learnt about curry howard correspondence ² and here we are =x.

Foreshadowing

Why Compile for FFIs

Doing so allows multiple programs to use the same library code at runtime, reducing memory usage and improving efficiency. Necessary compiler related tools features for it can be codegen ³, bindgen ⁴, Foriegn function intrecae, wrappers or the language bindings.

If the language ,functions of which, you export and one you want to import this dynamic shared library object to, feature allows it by design, natively or within doable data marshalling⁵ and runtime overhead.

To qualify my point with illustrations in Rust and pyO3

Meta Goal

Illustrating these steps to the data ops template with rust and well data =x

DIY visit the attached sourcehut links of snippets, for full workspace code.

Foreshadowing

1. Proc Macro - I

   1. Rust library

      To generate a companion builder struct with the same fields wrapped in Option<T> to track if values are set setter methods for each field on the builder struct, taking the value and storing it as Some(value), returning &mut self for chaining & build method to check and construct. Support optional fields in the original struct, allowing those to be omitted without error.

   2. A main function

      Use builer =pattern= to get a struct Command with fields executable, args, env, and current_dir & to derive the builder macro to generate CommandBuilder

Src for main the function is below checked and built

// fn main() {
//     println!("Hello, world!");
// }
//Option<String>
use builder_macro::Builder;

#[derive(Builder)]
pub struct Command {
    executable: String,
    args: Vec<String>,
    env: Vec<String>,
    current_dir: Option<String>,
}
// mathod chaining
fn main() {
    let command = Command::builder()
        .executable("cargo".to_owned())
        .args(vec!["build".to_owned(), "--release".to_owned()])
        .env(vec![])
        .current_dir("..".to_owned())
        .build()
        .unwrap();

    assert_eq!(command.executable, "cargo");
}

Using pyo3 to genrate API for Python types in Rust

Basically the lib below defines simple rust structs, classes, methods, functions etc to compile the dynamic shared library object, ie a native Python extension which you can import as a module in python client script to create instances of rust clasess exposed as python classes and then call those methods.

Src

/*
What did I learn
Bound<'py, T> is PyO3's primary smart pointer type for Python objects tied to the GIL lifetime 'py,
Giving  API for Python types in Rust.
For safer  GIL-bound usage and lifetime correctness, for thread safety
Implement the trait before you want to use it
For the lastest version
    */
use pyo3::prelude::*;
use pyo3::types::PyModule;

#[pyclass]
struct NumberList {
    numbers: Vec<i32>,
}

#[pymethods]
impl NumberList {
    #[new]
    fn new() -> Self {
        NumberList {
            numbers: Vec::new(),
        }
    }

    fn add(&mut self, value: i32) {
        self.numbers.push(value);
    }

    fn len(&self) -> usize {
        self.numbers.len()
    }

    fn clear(&mut self) {
        self.numbers.clear();
    }
}

#[pymodule]
fn ownership(m: Bound<'_, PyModule>) -> PyResult<()> {
    m.add_class::<NumberList>()?;
    Ok(())
}

python client

What next

1. ARM64 sbc builds for data processing, machine learning or RToS applications ⁶, requiring high efficiency .

Footnotes

Copyright (C) 2013-2025 Xameer . All rights reserved.