Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face

Google Cloud C4 Virtual Machine (VM) running on Intel® Xeon® 6 processors (codenamed Granite Rapids (GNR)) was benchmarked to evaluate improvements in the text generation performance of OpenAI GPT OSS Large Language Model (LLM). The results demonstrate a 1.7x improvement in Total Cost of Ownership (TCO) over the previous-generation Google C3 VM instances.

GPT OSS is an open-source Mixture of Experts (MoE) model. MoE models allow for efficient scaling of model capacity without linearly scaling compute costs by using specialized “expert” sub-networks. Intel and Hugging Face collaborated to optimize expert execution (PR #40304), eliminating redundant computations and improving utilization.

Benchmarking was conducted using the unsloth/gpt-oss-120b-BF16 model with bfloat16 precision. Across batch sizes up to 64, the Intel Xeon 6-powered C4 consistently outperformed the C3 with a 1.4x to 1.7x throughput per-vCPU. At batch size 64, C4 provides 1.7x the per-vCPU throughput of C3, yielding a significant TCO advantage. These results underline that large MoE models can be efficiently served on next-generation general-purpose CPUs thanks to targeted framework optimizations.