How HN: Ironkernel – Python expressions, Rust parallel

Write NumPy-like expressions in Python. Execute them in parallel on Rust, outside the GIL.

Rayon uses all CPU cores automatically. Go-style channels and select enable concurrent pipelines.

Python DSL Rust engine
------------------------------ ---------------------------------
kernel.arg / @kernel.elementwise build IR / KernelSpec / MapSpec
rt.go(...) release GIL and execute in rayon
task.result() return Buffer / scalar result
chan / select bounded channel handoff

Install into the Python environment you will run the code with using python -m pip. Python 3.9+ and NumPy 1.24+ are required.

python -m pip install --upgrade pip
python -m pip install ironkernel

Verify the installation:

python -m pip show ironkernel
python -c "import ironkernel; print('ironkernel', ironkernel.__version__)"

The ironkernel._ironkernel extension installs as a prebuilt wheel whenever a compatible OS/Python combination exists. On systems without a matching wheel, pip falls back to source build, which requires a working Rust toolchain.

Basic Usage Example:

import numpy as np
from ironkernel import kernel, rt
# 1) Prepare input buffers
a = rt.asarray(np.array([1.0, 2.0, 3.0], dtype=np.float64))
b = rt.asarray(np.array([10.0, 20.0, 30.0], dtype=np.float64))
# 2) Define computation with the DSL
@kernel.elementwise
def add(x, y):
    return x + y
# 3) Convert to execution spec
spec = kernel.map(add, x=a, y=b)
# 4) Execute (GIL is released inside Rust runtime)
task = rt.go(spec)
out = task.result() # Blocks until task completion
arr = out.numpy()
print(arr) # [11. 22. 33.]

Concurrent Pipelines:

from ironkernel import chan, kernel, rt
@kernel.elementwise
def double(x):
    return x * 2.0
src = rt.asarray(np.arange(6, dtype=np.float64))
out_ch = chan(4)
task = rt.go(kernel.map(double, x=src), out=out_ch)
buf = out_ch.recv()
print(buf.numpy()[:3])
task.result()

Reductions and Conditionals:

Ironkernel supports kernel.sum, kernel.mean, kernel.min_reduce, and kernel.max_reduce. The kernel.where function provides lazy evaluation for conditional branches.

Quality Assurance:

The project enforces 100% statement and branch coverage for the Python layer. Rust code is instrumented for coverage and subjected to stress tests and mutation testing via cargo-mutants to ensure reliability in concurrent environments.