Gaia2 and ARE: Empowering the community to study agents

That’s why we’re very happy to introduce Gaia2, the follow-up to the agentic benchmark GAIA, allowing analysis of considerably more complex behaviors. Gaia2 is released with the open Meta Agents Research Environments (ARE) framework to run, debug and evaluate agents. ARE simulates complex real world-like conditions and can be customized to further study agents behaviors. Gaia2 dataset is released under CC by 4.0 license, and ARE under MIT license.

GAIA is an agentic benchmark published in 2023, with 3 levels of information retrieval questions requiring tools, web browsing, and reasoning to solve. In 2 years, the easiest levels have become too easy for models, and the community is coming close to solving the hardest questions, so it was time for an entirely new and harder agent benchmark!

Here comes Gaia2, a follow up to GAIA, going way beyond it in terms of capabilities studied!

Where GAIA was read-only, Gaia2 is now a read-and-write benchmark, focusing on interactive behavior and complexity management. Agents are now evaluated not only on search and retrieval, but also on instruction following over ambiguous or time-sensitive queries, in a noisy environment with controlled failures - reflecting real-world conditions more than any other simulated environment. We want to test how agents manage tools or APIs that sometimes do not work, plan successions of actions with very specific time frames, and adapt to new events - a whole new range of complexity!

To do this, we use the following task groups (thanks to 1000 brand new human-created scenarios):

Execution: Multi-step instruction following and tool-use (e.g., contact updates)Search: Cross-source information gathering (e.g., friend cities from WhatsApp)Ambiguity Handling: Clarification of conflicting requests (e.g., scheduling conflicts)Adaptability: Response to changes in the simulation (e.g., updating an email using follow up information)Time/temporal Reasoning: Time-sensitive actions (e.g., cab orders after 3-minute delays)Agent-to-Agent Collaboration: Communication between agents without direct API accessNoise Tolerance: Robustness to API failures and environmental instability

In the spirit of GAIA, scenarios do not require specialized knowledge: humans should in principle be able to get 100%, which allows easy debugging for model developers.

Gaia2 runs with ARE, an execution environment, where an agent of your choice has access to a combination of applications and associated pre-populated data.

For Gaia2, we created a smartphone mock-up environment, simulating what a human would use in their daily life. It contains real-world applications such as messaging (Email), utilities (Calendar, Contacts, Shopping, a FileSystem, …), and a chat interface to talk to the agent. All applications are also accessible to the agents through tool calling. Last but not least, the demo also contains a simulated persona’s history of conversations and app interactions.

All agent interactions are automatically recorded as structured traces during execution for deep dives and analysis: they include tool calls, API responses, model thoughts, timing metrics (e.g., response latency), user interactions, and so forth - and can all be exported as JSON.

For reference, we compare a range of large open and closed source models: Llama 3.3-70B Instruct, Llama-4-Maverick, GPT-4o, Qwen3-235B-MoE, Grok-4, Kimi K2, Gemini 2.5 Pro, Claude 4 Sonnet, and GPT-5 in all reasoning modes.

All models are evaluated using the same setup (a uniform ReAct loop for consistency, temperature of 0.5, generation limit of 16K tokens), with a combination of model-as-a-judge (Llama 3.3 Instruct 70B) and exact-match evaluation depending on the particular task. All 101 tools (and the general environment description) are provided in the system prompt.

Among the evaluated models, the highest-scoring model overall as of September 2025 is GPT-5 with high reasoning, and the best open source model is Kimi K2.

Some capabilities appear to be already close to solved by the best models: execution of simple tool calls and instruction following (execution), and overall search (as we could have guessed from current results on GAIA). The ambiguity, adaptability, and noise splits remain challenging for now for all models, and it’s interesting to see that performance on what were considered complex agentic tasks (instruction following and search) is not a good proxy for performance on closer-to-real-world tasks. Last but not least, the hardest split for all models at the moment is the time one: it’s very hard at this moment for models to correctly handle time-sensitive actions (though this could likely be mitigated by the use of specialised tools and better temporal reasoning). Detailed analysis of these results can be found in the paper.

However, we believe it’s important to push reporting beyond raw scores: if the model is correct but took several thousand tokens to reach the correct solution, or ran for several hours, it is “not as good” as a model which succeeded orders of magnitude faster. We therefore also normalize scores for cost, quantified as the average number of LLM calls and output tokens (which both define a cost-performance Pareto frontier). In the paper you’ll find score vs monetary cost and time.

If you want to evaluate your model on Gaia2, you can follow these steps:

First, install Meta's Agent Research Environment in your Python environment of choice (uv, conda, virtualenv, ...)

pip install meta-agents-research-environments

Then, run the benchmark for all configurations: execution, search, adaptability, time and ambiguity. Don't forget to upload all results to the hub with the hf_upload kwarg!

are-benchmark run --hf meta-agents-research-environments/Gaia2 --split validation --config CONFIGURATION --model YOUR_MODEL --model_provider YOUR_PROVIDER --agent default --max_concurrent_scenarios 2 --scenario_timeout 300 --output_dir ./monitored_test_results --hf_upload YOUR_HUB_DATASET_TO_SAVE_RESULTS

Run the oracle to get your aggregated score file

are-benchmark judge --hf meta-agents-research-environments/Gaia2 --split validation --config CONFIGURATION --agent default --max_concurrent_scenarios 2 --scenario_timeout 300 --output_dir ./monitored_test_results --hf_upload YOUR_HUB_DATASET_TO_SAVE_RESULTS

Finally, add all the relevant information about your model in the README, and share it on the leaderboard to centralize Gaia2 traces here!

Beyond benchmark scenarios, you can use Gaia2 apps and content in ARE to see if the model is able to correctly solve less verifiable tasks such as loading emails, writing follow-ups, adding events to the calendar or booking meetings - in sum, providing the perfect setup to evaluate your AI assistants through interaction!

You can also easily customise the environment, by 1) connecting your tools (via MCP or directly ) to test your agents on it; 2) implementing your own scenarios, including defining trigger or timed events (eg: after 2 minutes, the Mail app will receive a new email from Contact), to see how the agent is able to adapt to an evolving environment

(As the agents are by default json agents, they can’t mess up your machine, unless of course you connect them to external apps with unsafe rights. So, operate with caution when adding your own apps or using untrusted MCPs)

Here are several use cases that we’ve used ARE for:

Vibe-check any agent on real or simulated data, to study a variety of setups, with their own rules, tools, content, and verifications - Test agent tool calling and orchestration capabilities, either with local apps or MCP tools - Generate your own tool-calling trace to fine-tune tool calling models - Easily gather and reproduce existing agentic benchmarks