Why Machines Cannot Learn Mathematics, Yet

Nowadays, Machine Learning (ML) is seen as the universal solution to improve the effectiveness of information retrieval (IR) methods. However, while mathematics is a precise and accurate science, it is usually expressed by less accurate and imprecise descriptions, contributing to the relative dearth of machine learning applications for IR in this domain. Generally, mathematical documents communicate their knowledge with an ambiguous, context-dependent, and non-formal language. Given recent advances in ML, it seems canonical to apply ML techniques to represent and retrieve mathematics semantically. In this work, we apply popular text embedding techniques to the arXiv collection of STEM documents and explore how these are unable to properly understand mathematics from that corpus. In addition, we also investigate the missing aspects that would allow mathematics to be learned by computers.

💡 Research Summary

The paper investigates why contemporary machine learning (ML) techniques, particularly text embedding methods, fail to adequately understand and retrieve mathematical content. Starting from the observation that mathematics, while precise, is communicated through ambiguous, context‑dependent, and non‑formal language, the authors argue that this mismatch hampers the application of standard information retrieval (IR) approaches.

In the background section, the authors review existing Mathematical Information Retrieval (MIR) methods. Most current systems attempt to link identifiers in formulas to textual definitions, relying on standards such as Content‑MathML. However, the specifications are vague: csymbol and ci elements are not consistently used, and there is no unified way to declare whether a token represents a function, a variable, or a composite identifier. Consequently, automatic parsing and semantic linking become unreliable. Prior work using Support Vector Machines, part‑of‑speech tags, and distance‑based scoring achieves only modest precision (≈48 %) and recall (≈28 %).

The paper then surveys modern embedding techniques: word2vec’s skip‑gram and CBOW, paragraph vectors (Distributed Memory and Distributed Bag‑of‑Words), multi‑sense embeddings, and the incorporation of lexical resources such as WordNet, ConceptNet, and BabelNet. While these methods have dramatically improved many NLP tasks, they ignore the structural relationships inherent in mathematical expressions.

Three main strategies for embedding mathematics are examined. (1) Single‑token embeddings (e.g., EqEmb) treat an entire formula as one token. This allows similarity measurement between whole equations but discards the internal semantics of symbols and sub‑expressions. (2) Token‑stream embeddings linearize a formula into a sequence of identifiers and operators. This captures all symbols but suffers from long token chains, limited context windows, and increased noise, especially for complex expressions. (3) Semantic‑group embeddings propose to first identify meaningful parts (e.g., function calls) and map them to unified groups before training, but such a preprocessing system does not yet exist.

Empirical evaluation on the arXMLiv 2018 dataset shows that while embedding‑based distance metrics can differentiate between equations, they do not learn connections between identifiers and their natural‑language definitions. For instance, the identifier “W(2,k)” is not linked to the textual phrase “Van der Waerden number” because the training data lacks explicit labeled pairs and the underlying MathML representation does not unambiguously encode the function’s meaning.

The authors identify four essential requirements for machines to truly learn mathematics: (1) a precise, standardized metadata layer for mathematical symbols and functions (e.g., an extended OpenMath dictionary); (2) large‑scale, high‑quality labeled datasets that map identifiers to definitions; (3) model architectures that ingest structural representations such as Syntax Layout Trees rather than flat token sequences; and (4) a dedicated mathematical knowledge graph that captures relationships among concepts. They argue that only when these components are jointly developed can ML models move beyond superficial similarity and support advanced MIR tasks, automated reasoning, plagiarism detection, and intelligent tutoring systems.

In conclusion, the paper demonstrates that current word‑embedding approaches are fundamentally ill‑suited for mathematical semantics due to ambiguity, lack of standardized structural encoding, and insufficient labeled data. It calls for a concerted effort to build robust mathematical ontologies, annotated corpora, and structure‑aware learning models, thereby paving the way for machines to finally “learn” mathematics.