BigCodeArena: Judging code generations end to end with code executions

BigCodeArena-- the first human-in-the-loop platform for evaluating code generation models through execution.

Inspired by LMArena for LLMs, we've built a platform that allows anyone to compare code generation models side-by-side, but with a crucial difference: you can actually run the code and see what it produces. Just submit a coding task, watch two different models generate solutions, execute both programs, and vote on which model produced better results. The outcomes are organized into a leaderboard that displays the community's highest-rated models.

The field of code generation has long struggled with reliable evaluation methods. Traditional benchmarks like HumanEval test code against predefined test cases, but these represent only a tiny fraction of real-world programming tasks. Human evaluation platforms exist for general chatbots, but they fall short for code: reading raw source code and mentally simulating its execution is cognitively demanding and error-prone, especially for longer programs or complex UI applications.

Consider this scenario:

You ask two AI models to build a responsive photo gallery website. Both generate code that looks syntactically correct. But which one is actually better? Without running the code, it's nearly impossible to tell. One might produce a beautiful, functional grid layout, while the other might have subtle bugs or poor styling that only become apparent when rendered in a browser.

This observation led us to a key insight: execution feedback is essential for humans to judge code quality reliably. That's exactly what BigCodeArena provides.

BigCodeArena extends the Chatbot Arena framework with powerful features specifically designed for code evaluation:

Every code snippet generated by models is automatically executed in isolated sandbox environments. Whether it's a Python script, a React web app, a PyGame game, or a C++ algorithm, you can see the actual output, not just the source code.

We currently support 10 languages (Python, JavaScript, TypeScript, HTML, C, C++, Java, Go, Rust, and Markdown) and 8 execution environments:

Web Frameworks: React, Vue, Core Web (vanilla HTML/CSS/JS)Python Frameworks: Streamlit, Gradio, PyGameDiagrams: MermaidGeneral Purpose Interpreters: Python and JavaScript code interpreters, plus compiled language runners

Unlike static code comparison, you can actually interact with the generated applications:

• Click buttons and test UI elements in web apps
• Play the games generated by models
• Edit the code and re-run it to test modifications
• View visual outputs like plots, charts, and diagrams

Real programming isn't one-and-done. BigCodeArena supports multi-turn interactions, allowing you to refine requirements, ask for features to be added, or request bug fixes -- just like working with a real coding assistant.

Since launching in February 2025, BigCodeArena has collected over 14,000 conversations from more than 500 unique users, with 4,700+ high-quality preference votes comparing 10 frontier LLMs.

Our users have explored remarkably diverse coding scenarios:

Web Design (36%): Building responsive websites, interactive dashboards, and web applicationsProblem Solving (23%): Algorithms, data structures, and computational challengesGame Development (16%): Creating interactive games with physics, collision detection, and graphicsScientific Computing (14%): Data analysis, visualization, and numerical simulationsCreative Coding (8%): Artistic visualizations, generative art, and experimental interfacesDiagram Creation (3%): Flowcharts, system architectures, and data visualizations

Python dominates with over 4,000 conversations, followed by JavaScript/TypeScript (3,359), HTML (1,601), and C++ (642). Among frameworks, direct Python interpreters lead usage (6,000 sessions), with React (2,729), Core Web (1,574), Streamlit (1,254), and PyGame (1,087) also seeing heavy use.

Most interactions are focused and efficient: 76% of conversations consist of just 2 turns (one request, one response), with a mean conversation length of 4.12 messages. However, the platform supports extended multi-turn debugging sessions when needed, with some conversations exceeding 10 turns as users refine complex applications.

From our 14K conversations, we filtered for high-quality pairwise comparisons: conversations with at least two turns and actual code execution. This yielded 4,731 voting samples, with each evaluated model receiving at least 700 votes. We aggregate these votes into Elo ratings using the Bradley-Terry model, which estimates the probability that one model beats another based on head-to-head comparisons.

To ensure robust rankings, we use 100 bootstrap resamples to construct 95% confidence intervals, so we can identify statistically significant performance differences between models.

We evaluate models under three settings to control for different factors:

All Data: Uses all pairwise comparisons regardless of execution environment or programming languageEnvironment Matched: Only compares models when both were executed in the same sandbox (e.g., both in React or both in PyGame)Language Matched: Further restricts comparisons to the same programming language

Rankings remain remarkably consistent across all three settings, revealing clear performance tiers:

Top Tier: o3-mini and o1-mini consistently lead with the highest Elo ratings and tight confidence intervals. These models maintain top performance regardless of environment or language constraints, showing strong robustness across coding scenarios. Claude-3.5-Sonnet follows closely, particularly excelling when language is controlled.

Mid Tier: GPT-4o, o1, and Gemini-2.0-Pro/Flash form a competitive middle tier. GPT-4o shows some sensitivity to language matching, suggesting room for improvement in multilingual consistency.

Open Source Models: Qwen2.5 variants and Llama-3.3-70B lag behind frontier proprietary models, highlighting the performance gap that remains between leading closed and open models.

Breaking down performance by programming language reveals interesting patterns:

• Top-tier models like o3-mini and o1-mini achieve dominant win rates in mainstream languages like Python, Java, and C++
• Gemini-2.0-Pro shows particular strength in Rust, achieving the highest win rate in that category
• Different models exhibit distinct areas of expertise, with frontier models excelling in different niches
• Open models like Qwen2.5 variants show inconsistent performance, particularly struggling with Rust and Go

Analyzing win rates by execution environment reveals how models handle different runtime contexts:

Robust Performers: o3-mini maintains consistently strong performance across React, Streamlit, Gradio, Core Web, and PyGame, demonstrating excellent environmental adaptability.

Stable but Selective: Claude-3.5-Sonnet and Gemini-2.0-Flash show generally stable performance but with reduced win rates in complex UI-heavy environments like Vue and Mermaid.

Framework-Specific Weaknesses: Qwen2.5 models, while competitive in some web frameworks (Core Web, React), struggle significantly with interactive and visualization-oriented environments like PyGame, Vue, and Mermaid. These environments often require precise handling of control flow, graphics rendering, and package dependencies.

These results highlight an important insight: aggregate Elo scores don't tell the whole story. Some models remain brittle under specific runtime constraints, and execution environment matters significantly for real-world deployment.