Finding a CPU Design Bug in the Xbox 360 (2018)

The recent reveal of Meltdown and Spectre reminded me of the time I found a related design bug in the Xbox 360 CPU – a newly added instruction whose mere existence was dangerous.

Back in 2005 I was the Xbox 360 CPU guy. I lived and breathed that chip. I still have a 30-cm CPU wafer on my wall, and a four-foot poster of the CPU’s layout. I spent so much time understanding how that CPU’s pipelines worked that when I was asked to investigate some impossible crashes I was able to intuit how a design bug must be their cause. But first, some background…

The Xbox 360 CPU is a three-core PowerPC chip made by IBM. The three cores sit in three separate quadrants with the fourth quadrant containing a 1-MB L2 cache. Each core has a 32-KB instruction cache and a 32-KB data cache.

Trivia: Core 0 was closer to the L2 cache and had measurably lower L2 latencies.

The Xbox 360 CPU had high latencies for everything, with memory latencies being particularly bad. And, the 1-MB L2 cache was pretty small for a three-core CPU. So, conserving space in the L2 cache in order to minimize cache misses was important.

CPU caches improve performance due to spatial and temporal locality. But sometimes temporal locality doesn’t actually happen. If you are processing a large array of data once-per-frame, it may be gone from the L2 cache by the time you need it again. You still want that data in the L1 cache, but having it in the L2 cache just evicts other data, slowing down the other cores.

Normally this is unavoidable. The memory coherency mechanism of our PowerPC CPU required that all data in the L1 caches also be in the L2 cache. The MESI protocol used for memory coherency requires that when one core writes to a cache line, any other cores with a copy of the same cache line need to discard it – and the L2 cache was responsible for keeping track of which L1 caches were caching which addresses.

But, for performance, a new instruction was added – xdcbt. The normal PowerPC dcbt instruction was a typical prefetch instruction. The xdcbt instruction was an extended prefetch instruction that fetched straight from memory to the L1 d-cache, skipping L2. This meant that memory coherency was no longer guaranteed.

I wrote a memory copy routine that optionally used xdcbt. A game developer reported weird heap corruption crashes. I realized that memory prefetched with xdcbt is toxic. If it is written by another core before being flushed from L1, two cores have different views of memory.

I fixed my routine, but then the same game started crashing again, even though it was no longer using the xdcbt instruction. I realized the problem was the branch predictor. The Xbox 360 CPU has long pipelines, making a branch predictor necessary. The branch predictor makes a prediction and the predicted instructions are fetched, decoded, and executed – but not retired until the prediction is known to be correct.

A speculatively-executed xdcbt was identical to a real xdcbt! Once a prefetch had been initiated, there was no way to cancel it. The branch predictor would sometimes cause xdcbt instructions to be speculatively executed and that was just as bad as really executing them.

We verified this by replacing every xdcbt in the game with a breakpoint. The breakpoints were not hit, proving the game was not executing the instructions, yet the crashes went away. This proved that instructions that were not executed were causing crashes. The instruction was too dangerous to have anywhere in the code segment of any game.