Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
Researchers have introduced Mask-Proof, an automated pipeline designed to evaluate the step-level mathematical reasoning of Large Language Models (LLMs). By converting real proofs into automatically checkable masked-step tasks, it bridges a critical gap in AI evaluation with expert-level accuracy.
Computer Science > Artificial Intelligence
Title: Mask-Proof: An LLM-based Automated Data Curation Pipeline on Mathematical Proofs
Abstract: Large language models (LLMs) are increasingly capable of mathematical problem solving and can even assist with research-level proofs, yet we still lack a scalable and reproducible way to measure step-level reasoning in long proofs across diverse sources. This evaluation gap limits trustworthy AI assistance in proof-certified scientific progress. Existing evaluations often emphasize final answers or rely on costly expert grading, while end-to-end proof generation remains open-ended and hard to verify automatically. We introduce Mask-Proof, a pipeline that turns real proofs into automatically checkable masked-step tasks. It masks key formula steps, provides the necessary surrounding context, and evaluates model reconstructions with an LLM-based equivalence judge using repeated votes for stability. The resulting Mask-ProofBench contains 292 curated problems across diverse research areas. Experiments with 17 models show that reasoning-enhanced models outperform standard models by 12% to 27%. Our evaluator achieves 96.8% agreement with expert annotators, enabling faithful, reproducible, and comparable measurement of step-level mathematical reasoning. Benchmark, annotations, and code are available publicly.
Source: arXiv cs.AI Recent
