SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

Researchers have introduced SEAGym, a new evaluation environment designed to accurately measure the self-evolution of LLM agents. This tool addresses the limitations of traditional evaluation methods, which often overlook overfitting or performance degradation.

Computer Science > Artificial Intelligence

Title:SEAGym: An Evaluation Environment for Self-Evolving LLM Agents

View PDF HTML (experimental)Abstract:Self-evolving LLM-based agents improve mainly by changing their agent harness: the structured execution layer around a base model, including prompts, memory, tools, middleware, runtime state, and the model-tool interaction loop. Existing evaluations often reduce this process to isolated task scores or a single sequential curve, obscuring whether an update produces reusable improvement, overfits recent tasks, increases cost, or harms older behavior. We introduce SEAGym, an evaluation environment for measuring agent harness updates across training, validation, test, replay, and cost records. SEAGym turns Harbor-compatible benchmarks into dynamic self-evolution task sources with train batches, frozen update-validation, held-out ID and OOD transfer views, replay diagnostics, and saved snapshot and metric records. Instantiating SEAGym on Terminal-Bench 2.0 and HLE, we compare ACE, TF-GRPO, and AHE under a shared epoch/batch protocol. The results show that these evaluation views provide complementary signals about the evolution process: frequent updates may fail to improve held-out performance, useful intermediate snapshots may collapse later, and source diversity and model backend can affect harness reliability.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

Nothing from Something: Can a Language Model Discover 0?

A new study investigates whether AI language models can independently discover the mathematical concept of 'zero'. The findings reveal that while models cannot generalize this concept out-of-the-box, language pretraining reduces the required training examples by 50%.

agentic-systems

Closing the Feedback Loop: From Experience Extraction to Insight Governance in Verbal Reinforcement Learning

Researchers propose a three-layer architecture (rules, evidence, skills) to close the feedback loop in verbal reinforcement learning, solving the retention-forgetting dilemma for LLM agents in non-stationary environments.

agentic-systems

MapSatisfyBench: Benchmarking Satisfaction-Aware Map Agents through Behavior-Grounded Implicit Decision Factors

Researchers have introduced MapSatisfyBench, a new benchmark for evaluating LLM-based map agents. It shifts the evaluation focus from simple task completion to satisfying implicit user needs and optimizing real-world user satisfaction.

agentic-systems

MemTrace: Probing What Final Accuracy Misses in Long-Term Memory

While LLM agents increasingly maintain long-term memory, traditional evaluation methods fail to show how facts behave under changing conditions. The new MemTrace benchmark reveals that the primary bottleneck in AI memory is evidence utilization, not retrieval.

agentic-systems

Surrogate Assisted Pedestrian Protection Design via a Foundation Model Orchestrated Workflow

Researchers have developed the first foundation model-orchestrated workflow for crash safety design, reducing evaluation times from hours of conventional CAE simulations to mere seconds.

agentic-systems

SkillChain-Gym: A Benchmark for Reskilling-Aware Production-Inventory Control under Disruptions

SkillChain-Gym is a new benchmark for production-inventory control that integrates workforce reskilling and training dynamics under disruptions. It evaluates how training policies compete with production capacity and helps decision-makers balance operational efficiency and workforce resilience.

EXPLORE TOPICS