The Zen of Parallel Programming: The Posture of a Kernel

As the journey through parallel programming continues, I shift my attention toward GPU kernels, and what better place to continue than the HipKittens: Fast and Furious AMD Kernels paper.

A GPU kernel is a specialized function launched many times across the GPU. Each invocation follows the same general operation while receiving its own position and portion of the data. The kernel itself may appear simple, but its execution unfolds through threads, wavefronts, thread blocks, tiles, registers, memory, and the architecture responsible for holding all of that activity together. On the AMD hardware studied in the paper, threads are grouped into 64-thread waves, while multiple waves are scheduled together inside thread blocks on the GPU's compute units.

The hard truth is that it depends upon the threads that express it, the data that gives it something to transform, and the hardware that determines how that transformation can physically occur. Even when the same tile-based abstraction moves from NVIDIA to AMD, HipKittens finds that the algorithms used to express that abstraction must be redesigned for AMD's architecture. The abstraction may travel, but it cannot remain untouched by the conditions receiving it.

One of the paper's central contributions is an eight-wave ping-pong schedule designed to overlap computation with memory movement. Eight waves are placed inside a thread block, with two waves residing on each of four SIMD units. Within each pair, one wave performs matrix computation while the other prefetches the data needed for the next stage. They then exchange roles, moving back and forth between computation and memory rather than remaining permanently attached to either identity.

This brought me back to the “Posture” chapter of Zen Mind, Beginner's Mind, where Suzuki describes us as both independent and dependent, “not two, and not one.” The paired waves remain distinguishable. One is computing while the other is moving memory, and each carries its own temporary responsibility. Yet neither role can explain the complete activity by itself. The compute wave depends upon data arriving, while the memory wave performs work whose value becomes visible through the computation that follows.

What interests me most is that neither wave remains permanently attached to its role. One prepares while the other acts, and then they exchange positions. Their responsibilities are real, but temporary. The larger activity depends not only upon each wave performing its task, but also upon each wave being able to release that task when the relationship calls for something different. Maybe posture is not only the position we occupy, but the relationship we maintain while that position changes.

This brings me back to the more general lesson in An Introduction to Parallel Programming. Parallel activity requires enough independence for distinct work to happen at the same time, but enough dependence for those separate actions to remain part of one computation. Complete independence would leave us with isolated work that never becomes a whole. At the other extreme, if every participant depended upon every other participant before making progress, parallel movement would collapse into waiting.

Human beings may live somewhere within that same tension.

We encounter conditions before we fully understand them. We carry memories, and we inherit family patterns, expectations, fears, and sometimes possible predispositions toward addiction or depression. We may also experience events that teach the body and mind to hold a particular posture long after the original condition has passed. These experiences are real, and their influence should not be dismissed simply because they belong to the past.

A role that once protected us may later prevent us from moving. A way of seeing that once helped us survive may eventually make every new condition resemble the one that created it. We may begin to believe that because the system learned to act in one way, it must always execute through the same posture.

Human beings are not GPU kernels, and trauma cannot be reduced to an instruction schedule. What interests me is the possibility that a posture learned under one set of conditions may not be the only posture available when those conditions change.

HipKittens reminds me that even when the purpose of an abstraction remains recognizable, its expression may have to change when it enters different conditions. The answer is not to pretend that the new architecture is identical to the old one. Nor is it to discard everything that came before. The work is to understand what can travel, what must be reorganized, and what no longer belongs in the present execution.

Maybe looking within is partly an attempt to recognize which parts of us are still carrying out instructions formed under conditions that are no longer here. Not so that we can erase the lives or experiences before us, but so that their influence does not have to remain the only posture available to us.

The past remains part of the system, but it does not always have to remain its scheduler.