AGENTIC-SYSTEMSJune 4, 20261 min read7 views

SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models

SMAC-Talk is a natural language extension of the StarCraft Multi-Agent Challenge (SMAC), designed to evaluate how LLM agents communicate, coordinate, and build trust. This open benchmark opens new avenues for researching multi-agent cooperation and handling deceptive communication.

Computer Science > Artificial Intelligence

Title:SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models

View PDF HTML (experimental)Abstract:As LLMs become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty. We introduce SMAC-Talk, a natural language extension of the StarCraft Multi-Agent Challenge for evaluating LLM-based agents in cooperative multi-agent environments. The environment has several key features such as decentralized control, partial observability and long-horizon decision making. SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust. We use this communication channel to construct different evaluation scenarios, including settings with an embedded deceptive communicator that tries to disrupt and deceive allies through communication alone. We provide three agents for benchmarking using 4 models from the Qwen3.5 family and study how reasoning structure, memory and model scale affect coordination between agents. We release SMAC-Talk as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent settings.

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Align AI to Dynamic Human-AI Workflows

Current AI alignment methods fall short by relying on static representations of human preferences. A new paper proposes a paradigm shift toward "interactive and complementary alignment," allowing AI and humans to co-evolve within dynamic workflows.

agentic-systems

MemoHarness: Agent Harnesses That Learn from Experience

Researchers introduce MemoHarness, an adaptive harness optimization framework that enables AI agents to learn from their own executions. By analyzing performance and storing insights in a dual-layer experience bank, it dynamically adapts to new tasks, outperforming static configurations.

agentic-systems

How Artificial Intelligence LLM Engines Shape the Global Conflict Information Environment

A new study reveals how AI-powered answer engines are highly vulnerable to hallucination and manipulation when reporting on global conflicts. The lack of robust digital records allows malicious actors to exploit these systems through Generative Engine Optimization (GEO), posing a new threat of AI-driven information warfare.

agentic-systems

AI Agents Do Not Fail Alone:The Context Fails First

A new study validates context-engineering quality as an independent leading indicator of AI agent reliability, introducing an open-source evaluation infrastructure called ProofAgent-Harness to predict behavioral outcomes before deployment.

agentic-systems

Interpretable Language Model for Closed-Loop Type 1 Diabetes Control

Researchers have developed LLM-T1D, a novel approach combining reinforcement learning and large language models to automate insulin delivery for Type 1 Diabetes. The system not only outperforms traditional methods but also explains its decisions in plain language, addressing the 'black-box' challenge in medical AI.

agentic-systems

RegNetAgents: A Multi-Agent Framework for Cross-Network Regulatory Driver Identification in Cancer Genomics

Researchers have introduced RegNetAgents, an AI-oriented multi-agent framework designed to identify regulatory driver candidates across heterogeneous gene networks in cancer genomics. By integrating bulk tumor and single-cell data, the system offers a powerful tool for discovering therapeutic targets in breast and colorectal cancers.