Running Doom on Our Custom CPU and Going Viral

Two weeks ago we successfully ran(crawled) Doom on a CPU we built from scratch (and then posted a video that got a few million views). To be honest, I still can't believe it. What did we really do though? Well, we designed a custom CPU at the logic gate level, connected it to peripherals, adapted the DOOM source code to run on our machine, and deployed it onto an FPGA to run in real time. Before running Doom, we had only run simple programs that we had written, like Pong and Mandelbrot sets. Now we can run full published games, but getting here was a rather rocky road.

The Requirements

Starting off with our pipelined design in this post, we set our sights on more complex programs. Pong was great, but it was from the 70s. We wanted to jump into the 90s. However, there were two big problems in our way: memory and speed. Larger programs need larger memory and our design could only utilize the FPGA's BRAM, which was less than a megabyte. The basic shareware for Doom (doom1.wad) is 14 megabytes. That doesn't even include the memory required to run the program, just to store it. The second barrier is speed. While Doom is a breeze for modern PCs, it's a slog for our CPU. We simply need to go faster.

Thus, Liam and I decided to pursue each problem individually. Liam is currently laying the groundwork for out-of-order processing which will enable far greater parallelism and pipeline fulfillment tricks. I took on memory integration. While it may sound simple: just hook up an extra memory chip, the reality is far more complicated.

Memory Integration

In our original CPU design, memory was very clean. FPGA BRAM has 1 cycle latency and is very simple to interface with. This consistency meant that our pipelined processor never needed to stall for memory because the consistent delay was directly incorporated into our pipeline. Furthermore BRAM is granular, we can read and edit the memory word by word.

DDR3 memory on the other hand is slow, has variable latency, and wide bus widths. This makes memory operations more complicated and less predictable. It also is a lot slower. If every memory operation was sent to the DDR3 memory, the CPU would slow to a crawl. This is where cache comes in. Programs don't use all of the memory at all times, so the cache stores active regions of memory in BRAM to increase access speed. A well optimized cache can almost eliminate the added latency due to DDR3.

The Design

This version of the CPU uses a relatively standard 5 stage pipeline: Instruction Fetch, Decode, Register Read, Execute, Writeback. Furthermore, this design abstracts memory operations away into a unified interface to simplify the core pipeline stages.

Core Pipeline

The instruction Fetch stage tracks the program pointer and fetches the correct instruction from memory via the instruction Cache (ICache). Unlike the previous design, it internally handles redirection and stalling. With the addition of DDR memory, redirection becomes far more complicated. Previously memory had one cycle latency, so you could fetch an instruction every cycle and switch the memory request address the cycle a redirection request comes in.

However, with the new design, if the fetch stage receives a redirection request, the memory might already have a request in flight. We now need to track that the next response from memory is invalid and then request the correct address. After resolving this issue the fetch stage works flawlessly.

The Decode stage is simple. It takes the 32-bit instruction, breaks it up into constituent parts and saves it to an instruction bundle that is sent forward through the pipeline. Now, there were some unique issues with this stage as well, but I'll save that for the later sections

The Read stage was significantly modified. One issue with our previous register file was that it used combinational reads that slowed down our CPU. This version pipelined reads, which adds a cycle of latency but shaves down the critical path delay. Another change was Read-After-Write (RAW) hazard handling. RAW hazards are where you read from a register before a pending write goes through, giving you incorrect data. The previous read stage internally tracked the last few register writes that had passed through it. This was logically efficient but fragile and didn't always work. More rigorous testing showed that special cases made it fail to detect hazards. The new version relies on a Register Usage Map (RUM) computed by the surrounding core and fed into the stage. The RUM is a 32-bit map that tracks if a register is in use or not. This naturally scales to variable pipeline lengths without requiring any magic numbers. Using the RUM, the Read stage decides if it must raise a hazard flag and stall the previous stages or let the instruction through.

The Execute stage decides what to do with each instruction. It routes parts of the instruction to various components, calling the ALU for arithmetic, resolving branches, and talking to memory. When it resolves jumps, it sends a flush signal backwards through the pipeline to wipe incorrect stages. When it sees memory instructions, it drops into a specific memory wait stage that stalls until the memory responds.

The Writeback stage simply writes values back to the reg file.

Memory Handling

The core pipeline is honestly pretty standard, the memory is where it gets juicy. In an ideally pipelined CPU, the instruction fetch stage fetches a new word and the execute fetches new data every cycle. For contrast the DDR3 on our FPGA can return 2 words every 30-60 cycles. This is significantly slower than our desired throughput, so we need a cache. Actually we need two caches. The fetch and execute stages might request two different addresses in one cycle, so to serve both of them we need an Instruction Cache (ICache) and Data Cache (DCache) that operate independently

Cache

The ICache and DCache are relatively simple and quite similar to each other. In fact, the ICache uses the exact same underlying code as the DCache but with the write logic stripped out. Both caches are simple single way directly mapped caches. In the current iteration, they each use 2048 cache lines with 4 words each.

When a memory request is received, the cache isolates the word address (by removing the two lowest bits) then computes the cache index. The cache index is the lowest 10 bits of the word address. It then pulls that cache line from BRAM. The cache line is composed of 3 components: data, tag, status. The data is the 4 actual words of memory that the cache is tracking. The tag is the upper bits of the word address and guards against cache aliasing. Aliasing is where multiple addresses point to the same cache index and we need to make sure that we do not confuse them. The 2 status bits represent valid and dirty signals. Valid means that the line data is actually data and not initialization randomness and the dirty signal lets us know if the CPU has modified the line since it was fetched from memory. If the pulled cache line is valid and has a matching tag, the cache performs a simple read or read-modify-write depending on the operation without ever making a memory request.

However, if the tag does not match, the cache must reference the DDR3 memory. There are two possible states of a valid cache line, clean and dirty. If a line is clean, we can just read the new cache line from the memory and discard the current data. If it is dirty, the cache must first writeback the modified line and then request the new data. Considering the DDR3 latency can be 30-100 cycles, while a cache hit is only 2 cycles, this is very slow.

Frankly, this is an inefficient design. A better cache would use longer cache lines to improve storage density and spatial reuse, but the 4 word line width was chosen to simplify interfacing with the 128 wide MIG interface on the original FPGA board. Later we switched to