Precision over perception: Why architecture matters in benchmarking

In the world of hybrid cloud infrastructure, data is our most valuable currency. But for data to be valid and valuable, it must be taken in context. Recently, VMware published a blog post claiming that, based on a study conducted by Principled Technologies, VMware Cloud Foundation (VCF) 9.0 with vSphere Kubernetes Service (VKS) delivers a "5.6x pod density" advantage over Red Hat OpenShift.

At first glance, the number is striking. As any systems architect knows, however, the validity of a benchmark lies not in the result, but in the methodology. When we look under the hood of this study, we find a comparison that highlights a fundamental misunderstanding of scale and a missed opportunity for a true "apples-to-apples" evaluation.

1. The Geometry of the test: 300 versus 4

The most critical element of any experiment is controlling the variables. In this study, both platforms were given the same physical hardware (four Dell PowerEdge servers). The configurations were very different, however:

The main Principled Technologies report does not mention how many virtual worker nodes VKS was running. The companion "science behind the report" methodology document does. In the appendix (pt-pod-density.yaml, page 5), the cluster configuration specifies "replicas: 300" (p. 5).

VKS was running 300 virtual worker nodes across 4 physical hosts. That is 75 virtual machines (VMs) per server, each configured as a "best-effort-medium" VM class (2 vCPUs, 8 GB RAM, no resource reservations). OpenShift was configured as a bare-metal cluster with 4 worker nodes, 1 per physical server.

In the published results, VKS reached 42,000 pods across its 300 workers, averaging 140 pods per node, while OpenShift reached 7,400 pods across its 4 workers, averaging 1,850 pods per node. OpenShift’s per-node density is over 13 times higher than VKS’s. The 5.6x headline reflects how the environments were sized—300 nodes versus 4, not how efficiently either platform runs pods.

The methodology document also reveals a configuration asymmetry in maxPods settings. The YAML configuration sets VKS to a maxPods limit of 200 per node. OpenShift was set to 5,000 per node, well above the default of 250. Even so, the topology determined the outcome: VKS’s theoretical ceiling was 60,000 (300 nodes x 200 per node), while OpenShift’s was 20,000 (4 nodes x 5,000 per node). The test compared 2 fundamentally different architectures and reported only the aggregate total.

2. Queuing theory versus platform speed

The report also highlights "faster pod readiness" for VKS. To the observer, this is a basic principle of queuing theory rather than a software engineering breakthrough.

Attempting to board 1,000 people onto 4 buses creates a line. Boarding onto 300 minivans simultaneously makes the line disappear. By using 300 virtual nodes, VKS minimized scheduling contention. Conversely, by pushing 4 bare-metal nodes to their absolute stability limits, the test created a bottleneck that wouldn't exist in a properly architected production environment.

The pods used in this test (via kube-burner) consume minimal CPU and memory relative to production workloads. The queuing advantage of 300 nodes over 4 is real, but it applies only to empty scheduling slots, not to pods that actually consume resources.

3. The missing comparison: Red Hat OpenShift Virtualization

If the goal was to test how a Kubernetes application scales on top of a hypervisor, there is a standard OpenShift configuration for that: OpenShift on OpenShift Virtualization.

By running virtual OpenShift compute nodes using OpenShift Virtualization, a kernel-based virtual machine (KVM) hypervisor, we achieve the same architectural topology as VKS on ESXi. Curiously, this "like-to-like" comparison was omitted. We believe customers deserve to see how platforms perform when the playing field is level: virtual-to-virtual or bare-metal-to-bare-metal.

4. The anatomy of a synthetic workload

The "workload" in the kube-burner test is designed to stress the Kubernetes control plane rather than the data plane. These are synthetic probes, not real business applications. They are meant to see how many slots the orchestrator can fill before it stops responding or the nodes lose their heartbeat.

5. Benchmark versus business reality

In a real-world business scenario, pods are not just "slots" to be filled. A production pod delivering business value carries a significant "payload" of resource requirements, I/O, and networking. High pod density with empty pods proves how many "placeholders" the system can handle; it does not reflect how many paying customers or transactions your infrastructure can actually support.

While we are proud that OpenShift was able to produce 1,850 pods per node, we recognize that this is not reflective of real customer workloads, which can and will saturate a worker node based on actual resource consumption.