Rust Threads on the GPU

At VectorWare, we are building the first GPU-native software company. Today, we are excited to announce that we can successfully use Rust's std::thread on the GPU. This milestone marks a significant step towards our vision of enabling developers to write complex, high-performance applications that leverage the full power of GPU hardware using familiar Rust abstractions.

Execution models

CPUs and GPUs execute programs in fundamentally different ways. A CPU program begins on a single thread and spawns additional threads as needed. Each thread runs independently and the programmer controls when and how concurrency is introduced.

GPU programs work differently. A GPU program consists of one or more kernels. Each kernel is launched with many instances that run in parallel. Concurrency is not something the programmer introduces explicitly. It is inherent in the way GPU programs are run by the hardware.

This model works well for uniform workloads like matrix multiplication, image processing, and graphics rendering where every warp does the same thing to different data.

As GPU programs grow more sophisticated, developers use warp specialization to activate different parts of the same program on different warps concurrently.

Functions as programs

Most CPU programming models begin with a main function as the program's entry point. Because execution begins with exactly one thread, representing the program as a function makes sense: the function body describes the work performed by that single thread.

Surprisingly, most GPU programming models use a function as their entry point as well. The programmer writes the function as if it executes once but the hardware then launches it thousands of times in parallel. GPU kernels are functions that look like normal CPU functions but behave very differently.

This mismatch between programming model and execution model is part of the reason why GPU programming is so hard. A function that runs once has very different semantics from one that runs thousands of times in parallel, yet both the compiler and the programmer cannot easily infer this by looking at the code alone. In practice this makes the programmer responsible for manually upholding invariants such as correct indexing into shared data and avoiding races.

GPU programs written in Rust follow the same pattern and are modeled as functions. Consider the same GPU kernel written in Rust: the kernel requires unsafe and takes a *mut f32 raw pointer rather than a reference. Because the GPU runs thousands of instances of this function simultaneously and each instance receives the same pointer, there is no way to express this safely as a function using Rust's ownership model. Rust was designed around the CPU's execution model where fn main runs on a single thread and the language can enforce safety. The GPU's execution model is foreign to the language and the kernel boundary is treated like an FFI boundary: raw pointers, unsafe, and no compiler guarantees. While this works, ideally Rust's safety guarantees would extend to the GPU as well.

We could introduce new types and annotations to capture GPU-specific semantics, but that would create a new programming model that is separate from ordinary Rust. It would require programmers to learn new abstractions and write GPU-specific code. We want GPU code to look like ordinary Rust code that integrates natively with the Rust ecosystem.

Why support std::thread on the GPU?

Rust programs use two primary models for concurrency: futures and threads. In a previous post we demonstrated futures and async/await running on the GPU for the first time. However, when we brought Rust's std to the GPU we did not implement threads. It was unclear how to do so and we already had ergonomic concurrency via async/await for writing GPU-native apps.

Yet much of the Rust ecosystem is built around threads rather than futures. Widely used thread pools such as rayon, async runtimes like tokio, and many libraries for parallelism all depend on std::thread. Supporting threads unlocks a large portion of the existing ecosystem.

Why not map std::thread to GPU threads?

Within warps, GPUs have many threads (also called lanes). An obvious approach is to map each std::thread to one of them. But a GPU "thread" is not what a CPU programmer means by "thread." A GPU thread is a single lane within a warp, more analogous to a SIMD lane on a CPU than an independent execution context.

A CPU thread has its own stack, its own program counter, and can be independently scheduled. GPU lanes do not work this way. Lanes within a warp advance together in lockstep. Mapping std::thread to GPU lanes would violate the semantics that Rust expects. It would also be slow. When lanes within a warp take different branches, the GPU hardware masks off inactive lanes and can execute each path sequentially. This is called divergence. If thread::spawn() mapped to a lane, the spawned lane and the calling lane would be in the same warp running different code. The hardware might serialize them, negating any concurrency benefit.

Implementation

Supporting std::thread on the GPU is enabled by three key observations:

1. Warps can behave similarly to CPU threads. Each warp has its own program counter, its own register file, and can execute independently from other warps. The GPU's warp scheduler switches between warps to hide latency, much like an OS scheduler switches between CPU threads.
2. A GPU kernel does not need to be concurrent at launch. The default GPU execution model starts all warps simultaneously, but nothing requires them to do useful work right away. By starting with a single active warp and enabling others on demand, we recover the CPU's execution model: one thread of control that introduces concurrency explicitly.
3. Warp specialization is manual thread partitioning with no language support. GPU developers already assign different tasks to different warps. std::thread is the same concept behind a language-provided API with ownership, type checking, and lifetime enforcement built in.

Here is how it works. We map each std::thread to a GPU warp. When a kernel starts, only the first warp begins executing user code while others wait in an idle loop. When thread::spawn is called, an idle warp is woken up to execute the task.