DeepMath: A lightweight math reasoning Agent with smolagents

By Intel AI Software Group

DeepMath is an aligned math reasoning agent built on Qwen3-4B Thinking and fine-tuned with GRPO (Group Relative Policy Optimization). Instead of verbose text, the model emits tiny Python snippets for intermediate steps, runs them in a secure sandbox, and folds the results back into its reasoning, reducing errors and output length. The agent is implemented using the smolagents library.

We evaluate DeepMath on four math datasets: MATH500, AIME, HMMT, and HLE, and show that:

🤖 The math agent alone reduces output lengths by up to 66%, while often improving accuracy.

⚡ GRPO training improves the agent performance even further, in almost all benchmarks.

Large language models (LLMs) have advanced reasoning capabilities, but mathematical problem-solving remains challenging; chain-of-thought traces can be lengthy and prone to arithmetic mistakes. Recent works demonstrate that small models can reach strong performance, and other studies investigate tool use to improve reliability. What those papers generally do not emphasize is reducing trace verbosity or explicitly training models to prefer short, computation-oriented traces executed in a constrained, auditable environment.

We focused on two goals:

1. Offload deterministic computation to a safe executor.
2. Train models to prefer concise, computation-oriented traces over verbose text.

DeepMath tackles this by combining a small Python executor with a fine-tuned LLM, enabling concise, computation-driven reasoning. The model learns to generate short Python snippets, which are executed in a sandbox and reintegrated into the context. GRPO fine-tuning encourages this behavior by rewarding correctness and encouraging shorter outputs.

• Base model: Qwen3-4B Thinking.
• Executor constraints: sandboxed environment, allow-list of imported modules, per-snippet timeout.
• Inference: based on smolagents, a math agent was created. vLLM is used as the inference engine.
• Training: based on the GRPO trainer in TRL, we modified TRL's vLLM client and server to generate GRPO completions using our DeepMath agent.

Agent Interface: During inference, the model can output normal tokens or special agent calls containing Python snippets. Execution: Snippets run in a sandboxed environment with strict safety constraints (no file I/O, no network, timeouts). Design Goals: Concision, Determinism, Safety, and Interpretability.

We fine-tune the model using GRPO, a reward-based optimization that balances accuracy, code usage, and length reduction. We implemented linear temperature scheduling (T=1.2 → T=0.7) to balance exploration and stability during training.

Key Insight: DeepMath reduces output length by up to 66% while improving accuracy on challenging datasets. Offloading computation reduces arithmetic errors, while shorter outputs mean faster inference and easier interpretability.