(Never) Mind your ps and qs: Von Neumann versus Jordan on the Foundations of Quantum Theory

In two papers entitled “On a new foundation [Neue Begr"undung] of quantum mechanics,” Pascual Jordan (1927b,g) presented his version of what came to be known as the Dirac-Jordan statistical transformation theory. As an alternative that avoids the mathematical difficulties facing the approach of Jordan and Paul A. M. Dirac (1927), John von Neumann (1927a) developed the modern Hilbert space formalism of quantum mechanics. In this paper, we focus on Jordan and von Neumann. Central to the formalisms of both are expressions for conditional probabilities of finding some value for one quantity given the value of another. Beyond that Jordan and von Neumann had very different views about the appropriate formulation of problems in quantum mechanics. For Jordan, unable to let go of the analogy to classical mechanics, the solution of such problems required the identication of sets of canonically conjugate variables, i.e., p’s and q’s. For von Neumann, not constrained by the analogy to classical mechanics, it required only the identication of a maximal set of commuting operators with simultaneous eigenstates. He had no need for p’s and q’s. Jordan and von Neumann also stated the characteristic new rules for probabilities in quantum mechanics somewhat differently. Jordan (1927b) was the first to state those rules in full generality. Von Neumann (1927a) rephrased them and, in a subsequent paper (von Neumann, 1927b), sought to derive them from more basic considerations. In this paper we reconstruct the central arguments of these 1927 papers by Jordan and von Neumann and of a paper on Jordan’s approach by Hilbert, von Neumann, and Nordheim (1928). We highlight those elements in these papers that bring out the gradual loosening of the ties between the new quantum formalism and classical mechanics.

💡 Research Summary

The paper undertakes a detailed comparative study of the two foundational approaches to quantum mechanics that emerged in 1927: Pascual Jordan’s statistical transformation theory (often called the Dirac‑Jordan formulation) and John von Neumann’s Hilbert‑space formalism. Both frameworks aim to express conditional probabilities—“the probability of obtaining a value of one observable given a value of another”—but they do so on fundamentally different conceptual and mathematical grounds.

Jordan’s approach is rooted in a direct analogy with classical mechanics. He insists that a quantum problem be formulated in terms of canonically conjugate variables, the p’s and q’s, exactly as in Hamiltonian mechanics. In his “Neue Begründung” papers he introduces a transformation function ϕ(p,q) (and its complex conjugate) that links the eigenstates of one observable to those of another. The conditional probability that an observable B takes the value b when observable A has the value a is given by the absolute square |ϕ(a,b)|². This rule, first stated in full generality by Jordan in 1927b, is essentially the modern Born rule, but Jordan derives it from the requirement that the transformation be unitary and that the p‑q pair be complete. The entire formalism is therefore built on the existence of a set of canonical variables and on the preservation of the classical phase‑space structure.

Von Neumann, working independently, rejects the need for any specific p‑q pair. He treats every observable as a self‑adjoint operator on a separable Hilbert space. The central object is a maximal set of commuting operators; their common eigenvectors form a complete orthonormal basis. Conditional probabilities are defined via projection operators: if P_A(a) projects onto the eigenspace of A with eigenvalue a, and the system is in state |ψ⟩, then the probability of obtaining a is ⟨ψ|P_A(a)|ψ⟩, and the conditional probability of b given a is ⟨ψ|P_A(a)P_B(b)P_A(a)|ψ⟩ / ⟨ψ|P_A(a)|ψ⟩. In his 1927a paper von Neumann restates the same “square‑modulus” rule but in terms of inner products of state vectors and projection operators, and in a later 1927b paper he attempts to derive it from more primitive axioms such as countable additivity, continuity, and the spectral theorem.

The paper also revisits the 1928 joint work of Hilbert, von Neumann, and Nordheim, which re‑expresses Jordan’s transformation theory within the Hilbert‑space language. Their analysis shows that Jordan’s ϕ(p,q) is mathematically equivalent to the kernel of a unitary operator that maps one eigenbasis to another; the choice of a particular p‑q pair corresponds merely to a particular choice of basis in the abstract Hilbert space. Consequently, the “canonical” variables are not indispensable for the probability calculus; they are a convenient representation when one wishes to retain a classical‑mechanics intuition.

The authors argue that Jordan’s insistence on canonical variables reflects a “conservative” stance, trying to preserve as much of the classical formalism as possible during the turbulent early years of quantum theory. Von Neumann’s “radical” stance, by contrast, embraces the abstract algebraic structure of operators and the geometry of Hilbert space, thereby freeing quantum mechanics from any lingering classical scaffolding. This philosophical divergence is mirrored in the technical details: Jordan’s derivations rely on the existence of a complete set of conjugate variables, while von Neumann’s proofs depend only on the spectral properties of commuting operators.

The paper concludes that the gradual loosening of ties to classical mechanics—exemplified by the shift from p‑q based formulations to operator‑centric ones—was not merely a matter of mathematical convenience but a decisive conceptual evolution. It paved the way for later developments such as quantum information theory, modern measurement theory, and the rigorous statistical mechanics of quantum systems, all of which rest on von Neumann’s abstract framework. The historical juxtaposition of Jordan and von Neumann thus illuminates how two competing visions, each initially plausible, resolved in favor of the more general, basis‑independent formalism that underlies contemporary quantum physics.