AGENTIC-SYSTEMSJuly 24, 20261 min read

JAXBench: Benchmarking Autonomous TPU Kernel Optimization

Researchers introduce JAXBench, a TPU-native benchmark suite designed to evaluate AI-generated kernel optimization. Testing shows that providing TPU-specific documentation enables Gemini 3 Flash to achieve up to a 1.36x speedup over the default XLA compiler.

Computer Science > Artificial Intelligence

Title:JAXBench: Benchmarking Autonomous TPU Kernel Optimization

View PDF HTML (experimental)Abstract:Rigorous benchmarks have driven progress in autonomous GPU kernel performance optimization by establishing a shared target to hillclimb on, but no equivalent exists for TPUs. We present JAXBench, a TPU-native benchmark suite for AI-generated kernel optimization on Google Cloud TPUs. JAXBench comprises 50 JAX workloads that are both relevant and provide headroom for optimization. We extract 17 production ML operators from architectures in the public MaxText library such as Llama-3.1, DeepSeek-V3, Mixtral, Mamba-2, and AlphaFold2, and translate 33 operators from KernelBench that are validated for correctness and set with new problem sizes that achieve high TPU v6e MXU utilization. Eight of the 17 production operators ship with hand-optimized Pallas kernels from the public Tokamax library and block-size tuned to establish an expert upper-bound baseline. We evaluate four feedback-driven methods on generating candidate Pallas kernels for JAXBench. Across the full suite with Gemini 3 Flash, we find that target-specific context matters more than model scale on a sparsely-documented DSL like Pallas. Conditioning on curated TPU documentation raises per-sample correctness from 5.8% to 37.3% and solves 48 of 50 benchmarks at a 1.28x geomean speedup. Search structure yields significant gains once correctness is achieved, with Autocomp's beam-search pipeline reaching a 1.36x geomean speedup over XLA. On the 8 hand-tuned kernels, Autocomp reaches 1.60x geomean over XLA, recovering most of the 2.08x Tokamax upper bound but trailing on the specialized paged and ragged attention operators. High-quality TPU kernel optimization remains a challenging task, and we release the JAXBench benchmark, evaluation harness, and baseline results to support open source contributions.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

InferenceBench: A Benchmark for Open-Ended LLM Inference Optimization by AI Agents

InferenceBench is a new benchmark evaluating AI agents on open-ended LLM inference optimization tasks under realistic constraints. While agents outperform default setups, qualitative analysis reveals key limitations in their strategic exploration capabilities.

agentic-systems

PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails

Researchers have introduced PersonaTrail, a novel benchmark evaluating personalized web agents through realistic browsing histories. Alongside it, the PACMem memory framework structures raw browsing data to significantly boost agent performance in inferring user preferences.

agentic-systems

Beyond Liars' Bench: The Impact of Lie Typology, Depth, and Sparsity on Deception Detection in LLMs

A new study reveals that deception detection in LLMs heavily depends on lie typology and training data structure, highlighting why current detection probes struggle in out-of-domain scenarios.

agentic-systems

Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment

A new benchmark evaluates five leading large language models on multi-sensor physical hazard assessments, revealing a critical vulnerability when evaluating combined sensor risks below individual thresholds. Despite near-perfect accuracy on single-sensor breaches, all models failed to issue precautionary signals for multi-sensor hazards.

agentic-systems

DC-Leap: Training-Free Acceleration of dLLMs via Draft-Guided Contiguous Leaping Decoding

Researchers have introduced DC-Leap, a training-free framework that accelerates inference in Diffusion Large Language Models (dLLMs) by up to 105x when combined with KV-Cache while maintaining generation quality.

agentic-systems

PlanE: Meta Planning of Data, Tuning, and Inference for Extractive-based LLMs

Researchers introduce PlanE, a novel framework featuring a Data-Tuning-Inference (DTI) planner designed to optimize data, tuning, and inference processes for extractive-based LLMs while reducing annotation costs.