AGENTIC-SYSTEMSJuly 23, 20261 min read

FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

FineServe provides an in-the-wild, multi-model LLM serving dataset collected from a global commercial marketplace to characterize real-world workload dynamics. It offers a specialized workload generator to help evaluate routing, scheduling, and capacity planning in multi-model LLM systems.

Computer Science > Artificial Intelligence

Title:FineServe: A Fine-Grained Dataset and Characterization of Global LLM Serving Workloads

View PDF HTML (experimental)Abstract:Large language models (LLMs) are increasingly deployed as always-on online services, making efficient LLM serving a critical systems challenge. Achieving low latency and high throughput under volatile demand requires deep understanding of real-world serving workloads, yet existing studies often rely on proxy traces or coarse-grained characterizations that fail to capture the heterogeneity of modern multi-model LLM platforms. We present FineServe, an in-the-wild, multi-model LLM serving workload dataset collected from a global commercial marketplace, enabling fine-grained characterization of real-world serving dynamics across heterogeneous models and tasks. Leveraging FineServe, we conduct a comprehensive analysis of arrival dynamics and token behavior, revealing fundamentally different fluctuation regimes across model architectures, scales and task intents. Building on these insights, we develop the FineServe workload generator, which composes fine-grained model-aware workloads into configurable mixtures tailored for benchmarking multi-model serving platforms. By exposing these fine-grained workload dynamics, FineServe provides a realistic foundation for evaluating routing, scheduling, and capacity-planning strategies in LLM serving systems. FineServe is available at this https URL.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Profile-Graph Memory for LLM Agents: Implicit Cross-Entity Traversal through Narrative Profiles

Researchers introduce ProGraph, a novel two-layer memory architecture designed to enhance multi-hop reasoning and precise recall for LLM agents. Alongside ProGraph, the team released MemHop, a new benchmark tailored for evaluating complex, multi-step memory retrieval.

agentic-systems

NEXUS: Structured Runtime Safety for Tool-Using LLM Agents

NEXUS is a novel structured-plan safety monitor designed for tool-using LLM agents. By combining deterministic rules and calibrated risk scoring, NEXUS enforces graded intervention policies with sub-millisecond latency.

agentic-systems

Information Discernment in Large Language Models

A new study introducing the Learn2Discern (L2D) benchmark reveals that large language models struggle to evaluate source reliability and truthfulness when integrating external knowledge. Researchers found that models often prioritize source popularity over reliability, though simple inference-time interventions can help mitigate these issues.

agentic-systems

OpenEvoShield: Dual Non-Stationary Continual Defense for Open-World Multi-Agent System Attacks

OpenEvoShield is a co-evolutionary continual defense framework designed to protect LLM-based multi-agent systems from dynamic injection attacks in open-world environments by addressing both fast attack evolution and slow normal behavior drift.

agentic-systems

Stochastic Primal-Dual Decoding for Multiobjective Generative Recommender Systems

Researchers introduced a lightweight inference-time decoding layer for generative recommender systems to optimize multi-objective slates without retraining. The stochastic primal-dual approach dynamically balances relevance and constraints, achieving a +1.8% gain in auxiliary objectives in real-world deployment.

agentic-systems

LISA: Linear-Indexed Sparse Attention for Efficient Long-Context Reasoning

Researchers introduce LISA, a plug-and-play attention module that reduces computational complexity from O(n^2) to O(nM), achieving a 50% inference speedup and a 5.6% accuracy gain on reasoning benchmarks for long-context models.