Data for Mathematical Copilots: Better Ways of Presenting Proofs for Machine Learning

The datasets and benchmarks commonly used to train and evaluate the mathematical capabilities of AI-based mathematical copilots (primarily large language models) exhibit several shortcomings and misdirections. These range from a restricted scope of mathematical complexity to limited fidelity in capturing aspects beyond the final, written proof (e.g. motivating the proof, or representing the thought processes leading to a proof). These issues are compounded by a dynamic reminiscent of Goodhart’s law: as benchmark performance becomes the primary target for model development, the benchmarks themselves become less reliable indicators of genuine mathematical capability. We systematically explore these limitations and contend that enhancing the capabilities of large language models, or any forthcoming advancements in AI-based mathematical assistants (copilots or ``thought partners’’), necessitates a course correction both in the design of mathematical datasets and the evaluation criteria of the models’ mathematical ability. In particular, it is necessary for benchmarks to move beyond the existing result-based datasets that map theorem statements directly to proofs, and instead focus on datasets that translate the richer facets of mathematical research practice into data that LLMs can learn from. This includes benchmarks that supervise the proving process and the proof discovery process itself, and we advocate for mathematical dataset developers to consider the concept of “motivated proof”, introduced by G. Pólya in 1949, which can serve as a blueprint for datasets that offer a better proof learning signal, alleviating some of the mentioned limitations.

💡 Research Summary

The paper “Data for Mathematical Copilots: Better Ways of Presenting Proofs for Machine Learning” offers a comprehensive critique of the current data and benchmark ecosystem used to train and evaluate AI‑driven mathematical assistants, especially large language models (LLMs). The authors observe that most widely‑used datasets—such as GSM8K, MA‑TH, and their variants—are “result‑centric”: they pair a problem statement with a final proof or answer, but they ignore the rich intermediate processes that real mathematicians employ, such as motivation, heuristic search, failed attempts, and strategic choices. This narrow focus limits models to memorising or pattern‑matching final solutions rather than learning how to discover and justify proofs.

A central theme is the emergence of a Goodhart‑law‑like feedback loop: as benchmark scores become the primary optimization target, researchers tune models to excel on these narrow metrics, while the benchmarks themselves become less reliable indicators of genuine mathematical ability. The authors highlight contamination risks—benchmarks leaking into training corpora—making reported high accuracies potentially illusory. Moreover, the datasets are heavily skewed toward elementary or competition‑style problems, neglecting advanced research‑level mathematics and the diverse modalities (natural language, LaTeX, diagrams) used by mathematicians.

To address these shortcomings, the paper proposes a three‑tier taxonomy of mathematical AI systems:

1. Specialized/Narrow models – domain‑specific tools that require formal input languages and are not self‑explanatory. They excel in narrowly defined tasks (e.g., elementary geometry solvers, inequality engines) and rely on highly structured symbolic data.
2. General‑purpose “Mathematical Copilots” – LLM‑based assistants that interact via natural language and images, can call specialized tools, and aim to support a mathematician’s workflow (literature search, conjecture generation, proof guidance). These require richer, multimodal training data that capture the full reasoning trajectory.
3. Fully automated “Universal” models – envisioned as autonomous theorem generators that can propose, prove, and refute statements with minimal human input. Realising this class demands massive, high‑quality research‑level datasets and robust verification pipelines.

The authors argue that progress toward the second tier (the true “copilot”) hinges on two key innovations in data:

• Motivated Proofs – drawing on George Pólya’s 1949 concept, datasets should include not only the final proof but also a narrative of how the proof was discovered: the initial intuition, heuristic choices, and why certain approaches were pursued. This “motivated proof” format provides a learning signal for models to emulate human‑like discovery processes.
• Process‑Centric Benchmarks – evaluation should move beyond a single correctness bit. Instead, benchmarks ought to grade each intermediate step, verify logical consistency with formal checkers, assess creativity, and possibly reward efficient search strategies. Hybrid systems that combine neural generation with symbolic verification (e.g., chain‑of‑thought prompting plus theorem‑assistant verification) exemplify this direction.

The paper also stresses the importance of rigorous dataset curation practices borrowed from librarianship and archival science: detailed metadata (source, difficulty, domain, licensing), documentation of ethical considerations, and transparent versioning to mitigate contamination and reproducibility issues. It notes that while end‑to‑end generation has succeeded in many ML domains, pure end‑to‑end proof generation is insufficient for mathematics because errors in intermediate steps invalidate the whole argument.

In summary, the authors call for a paradigm shift: from result‑only datasets to richly annotated, multimodal collections that encode the full epistemic journey of mathematical reasoning. By integrating motivated proofs, step‑wise verification, and robust data governance, future LLM‑based mathematical copilots can move beyond “answer‑machines” toward genuine thought partners capable of assisting real research. This shift is presented as essential for the next generation of AI‑augmented mathematics.