Soul Player C64 – A real transformer running on a 1 MHz Commodore 64

A real transformer running on a 1 MHz Commodore 64.

A 2-layer decoder-only transformer - the same architecture behind ChatGPT, Claude, and Gemini - implemented in hand-written 6502/6510 assembly and running on an unmodified Commodore 64. ~25,000 int8 parameters. Real multi-head causal self-attention, real softmax, real RMSNorm. About 60 seconds per token. The whole thing fits on a floppy disk with room to spare.

2 layers, 4 attention heads × 8 dims, 32-dimensional embeddings, 64 FFN hidden units. ~25,000 parameters quantized to int8 with per-tensor shift scaling. The key breakthrough was fixing the softmax score normalization - shifting attention scores by 14 bits instead of 17 gives the 128-entry exp lookup table enough dynamic range to produce meaningful attention weights.

Training & Deployment

This is the fun part. Write a corpus, train a model, build a floppy. Using PyTorch, you can train a BPE tokenizer (128 tokens) and a QAT transformer. The training loop evaluates the actual integer forward pass every 500 epochs and saves the best checkpoint by int8 accuracy.

Technical Specifications

| Feature | Specification | | :--- | :--- | | Vocab | 128 tokens | | Embedding | 32 dimensions | | Layers | 2 | | Attention | 4 heads × 8 dims | | Context | 20 tokens | | Parameters | ~25,000 (int8) | | Weight size | 25 KB |

Each layer follows the standard flow: RMSNorm → multi-head causal self-attention → residual → RMSNorm → ReLU MLP → residual. All activations are Q8.8 fixed-point (int16). Since the 6502 has no multiply instruction, everything is implemented using shift-and-add routines.

Limitations

It is not smart. 25K parameters is about 70 million times smaller than GPT-4. It will produce broken sentences. It is also slow, taking about 60 seconds per token on real hardware. However, it proves the architecture works even at this extreme scale.