AGENTIC-SYSTEMSJuly 14, 20261 min read

Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

A new study reveals that minor formatting changes in prompt wrappers can significantly alter LLM performance scores, potentially flipping leaderboard rankings. Researchers introduce two new metrics, FSI and PSI, to measure this sensitivity and offer crucial recommendations for benchmarking and structured-output deployments.

Computer Science > Artificial Intelligence

Title:Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

View PDF HTML (experimental)Abstract:Prompt wrappers often differ only in formatting, yet they can change model scores enough to flip leaderboard conclusions. We study this variance under a token-controlled protocol and introduce two complementary metrics: the Format Sensitivity Index (FSI), the accuracy range induced by wrapper choice, and the Parseability Sensitivity Index (PSI), the corresponding range in answer parseability. Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models from 7B to 72B parameters, we find that mean FSI varies by over 30x across models and is largely explained by compliance failures. A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. We argue that reporting accuracy without wrapper variance and compliance is statistically fragile, and we give practical recommendations for both benchmarking and structured-output deployments.

Current browse context:

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Faithful, Not Corrective: Message-Format Effects in Multi-Hop Agent Relays Are Tier-Dependent

A new study reveals that message formats in multi-hop LLM agent relays significantly impact data fidelity, especially for smaller models. While structured formats like JSON prevent information drift, they act as faithful, error-localizing channels rather than error-correcting codes.

agentic-systems

GES-TSP: Graph Edge Sparsification for TSP

Researchers have proposed GES-TSP, a learning-based graph edge sparsification method that accelerates solving large-scale Traveling Salesman Problems (TSP). This approach can prune up to 99% of unnecessary edges while maintaining near-optimal accuracy.

agentic-systems

The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation

The study introduces execution-gated self-distillation using a strict-launch filter to boost out-of-family game generation from 8.8% to 42.2% on GameCraft-Bench. It demonstrates that a precise, ungameable verifier acts as an effective curriculum for training code generators.

agentic-systems

Boltzmann MapReduce: A Partition-Function Reduce for Forkable Sandboxes

A new research paper introduces 'Boltzmann MapReduce', bridging statistical physics and distributed computing by modeling data chunks as thermodynamic systems to optimize the Reduce step.

NOW LET US Related – YUKTI: From Natural-Language Situations to Robust, Verifiable Decisions An Uncertainty-Typed Proposition IR, Assumption-Robust Pareto Frontiers, and a Regret Certificate

agentic-systems

YUKTI: From Natural-Language Situations to Robust, Verifiable Decisions An Uncertainty-Typed Proposition IR, Assumption-Robust Pareto Frontiers, and a Regret Certificate

Current LLM-based optimization pipelines are fragile due to their reliance on single-point assumptions. YUKTI addresses this by introducing an uncertainty-typed proposition graph and Assumption-Robust Pareto Frontiers, reducing decision regret by over 90% compared to naive approaches.

agentic-systems

From ML Predictions to Informed Diagnostic Assistance Using the Toulmin Model of Argumentation

Researchers have proposed a new approach based on the Toulmin model of argumentation to structure and interpret AI-based image diagnoses. By combining specialized models like MedGemma and MedSigLip, the framework allows medical experts to critically assess ML-generated diagnoses rather than accepting them at face value.