AGENTIC-SYSTEMSJuly 7, 20261 min read

MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

Researchers have introduced MedCalc-Pro, a new benchmark and agent framework designed to help large language models (LLMs) solve complex medical calculations in real-world clinical settings.

Computer Science > Artificial Intelligence

Title:MedCalc-Pro: Solving Complex Medical Calculations with LLM Agents

View PDF HTML (experimental)Abstract:Current benchmarks for evaluating large language models (LLMs) in medical calculation are largely based on simplified settings, where each patient case corresponds to a single calculator and the required tool is explicitly specified in the query. However, real clinical scenarios often require multiple calculators for joint evaluation, nested-scale calculation, and fuzzy queries that do not directly specify the target calculator. To this end, we propose a new medical calculation benchmark, MedCalc-Pro, which covers three progressively challenging task settings: single-calculator, multi-calculator, and nested-calculator calculation settings. MedCalc-Pro contains 2,268 real-world clinical cases, covering 77 medical calculators across 14 clinical departments. Meanwhile, to address the limited performance of existing frameworks and methods in complex clinical scenarios, we further propose a more generalizable agent framework that supports multi-tool selection and nested-tool calling, while suppressing parameter error propagation through structured validation and evidence review. We conduct systematic comparisons across open-source, closed-source, and medical-specialized LLMs, and the results show that our framework achieves the best performance across all three task settings. This work provides a new benchmark and method for evaluating and applying LLMs in challenging medical calculation scenarios.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

Stay in the loop

Get the best AI & tech insights in your inbox — no spam, unsubscribe anytime.

More in this category

agentic-systems

Automated Data Readiness for Scientific AI

Researchers have introduced REDI, an open-source framework that automates the transformation of raw scientific datasets into AI-ready data. Tested on the Frontier supercomputer, REDI addresses the data preparation bottleneck in scientific AI domains.

agentic-systems

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

Researchers have introduced ASK+, a novel framework that significantly enhances small language model (SLM) assistance for reinforcement learning agents under partial observability. By providing trajectory-aware context and structured reasoning, ASK+ demonstrates that prompt design and selective gating can outperform sheer model scale.

agentic-systems

When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions

Evaluating agentic policy repair without per-state expert labels is challenging, as aggregate behavioral alignment metrics can be highly misleading. A new study using a hotel-pricing simulator demonstrates that LLM-based editors can achieve near-benchmark revenue by leveraging indirect diagnostic feedback, highlighting the need for closed-loop evaluation.

agentic-systems

Embodied Operators and Benchmarking: Toward Reusable and Deployable Embodied Intelligence Systems

A new study proposes 'embodied operators' to modularize embodied intelligence systems, enabling reusable and deployable robot pipelines beyond end-to-end models, supported by a multi-dimensional benchmarking framework.

agentic-systems

Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita

Researchers introduce Incognita, a framework to evaluate generative agents in socially distributed task environments where knowledge is partitioned across multiple entities. Testing shows that while advanced models perform better at gathering hidden information, overall reliability in executing grounded actions remains low.

agentic-systems

Object-Centric Environment Modeling for Agentic Tasks

Researchers propose Object-Centric Environment Modeling (OCM) to help AI agents organize accumulated experiences into an executable programmatic model, addressing the limitations of traditional free-form textual memories.