Evaluating Numerical Accuracy in Mixed-Precision Computing by Dual-Delta Testing

Mixed-precision computing has become increasingly important in modern high-performance computing and machine learning applications. When implementing custom mixed-precision functions – such as fused operators, optimized GPU kernels, or quantized inference paths – it is critical to verify their numerical accuracy. Traditional approaches typically compare the custom implementation against a reference using a single error metric. However, this single-delta approach provides limited insight into whether the observed errors are inherent to the precision level or specific to the implementation. This paper introduces \textit{Dual-Delta Testing}, a systematic methodology that evaluates two error distributions against a high-precision oracle, enabling rigorous comparison between a custom implementation and a baseline reference. We present the mathematical framework, algorithmic formulation, statistical analysis techniques, and practical examples demonstrating the methodology’s effectiveness in evaluating numerical accuracy.

💡 Research Summary

The paper addresses the growing need for rigorous numerical‑accuracy validation of custom mixed‑precision functions, such as GPU kernels, fused operators, and quantized inference paths, which are increasingly prevalent in high‑performance computing and deep‑learning workloads. Traditional validation relies on a single error metric (Δ_single) computed between a custom implementation and a reference implementation. This “single‑delta” approach cannot distinguish whether observed discrepancies stem from the inherent limitations of the reduced‑precision format or from bugs in the custom code, leading to ambiguous conclusions about correctness.

To overcome this limitation, the authors propose Dual‑Delta Testing, a methodology that introduces a high‑precision oracle (fΩ) as a trusted ground truth. For each input sampled from a prescribed distribution, the error of the custom implementation (f₁) and the error of the reference implementation (f₂) are measured independently against the oracle, yielding two error distributions Δ₁ and Δ₂. By comparing these distributions statistically, one can determine whether the custom implementation achieves comparable accuracy, is more accurate, or is less accurate than the reference, and also assess relative numerical stability via variance comparison.

The mathematical framework defines implementations as functions f: X → Y, an oracle fΩ with substantially higher precision (e.g., FP64 for FP16/FP32 work, or arbitrary‑precision MPFR for safety‑critical domains), and an error metric ϵ: Y × Y → ℝ₊. Given N inputs {xᵢ} drawn from P(x), the algorithm computes Δ₁ = {ϵ(f₁(xᵢ), fΩ(xᵢ))} and Δ₂ = {ϵ(f₂(xᵢ), fΩ(xᵢ))}. Propositions formalize equivalence (indistinguishable distributions), superiority (significantly smaller mean), and stability (significantly smaller variance). The paper recommends non‑parametric hypothesis tests—Kolmogorov‑Smirnov for distributional equality, Wilcoxon signed‑rank or sign test for mean comparisons, and paired t‑test when normality can be justified—because error distributions in numerical computing are often skewed or heavy‑tailed.

Algorithm 1 details the testing loop: generate an input, evaluate f₁, f₂, and fΩ, compute the chosen error metric, and append the results to Δ₁ and Δ₂. The authors stress the importance of a robust input generator that mixes random sampling (Gaussian, uniform) with domain‑specific edge cases (e.g., sparse matrices, extreme activation patterns) to expose worst‑case numerical behavior.

Oracle selection must satisfy the precision‑separation condition ϵ(fΩ, f_true) ≪ ϵ(fᵢ, f_true) for all i, ensuring that oracle error is negligible relative to the implementations under test. For FP16 or BF16 implementations, FP64 or FP32 typically suffices; for more demanding applications, MPFR can be used despite higher computational cost.

Error‑metric choice is left flexible. The paper discusses norm‑wise relative error, maximum hybrid error (combining absolute and relative terms), and notes that the latter gracefully handles near‑zero outputs. The selected metric influences the interpretation of Δ₁ and Δ₂ and should align with the target application’s tolerance criteria.

Statistical analysis proceeds in three stages. First, descriptive statistics (mean, median, standard deviation, selected percentiles, maximum) provide a quick sense of central tendency, spread, and tail behavior. Second, visualizations—overlapping histograms, box plots, Q‑Q plots, and scatter plots of Δ₁ versus Δ₂—reveal distribution shape, overlap, and correlation of errors across inputs. Third, formal hypothesis testing confirms whether observed differences are statistically significant.

A concrete case study evaluates FP16 matrix multiplication on a GPU (custom kernel f₁) against a CPU reference implementation (f₂), using FP64 matrix multiplication as the oracle (fΩ). With 128 × 128 matrices and 1,000 random test cases, the authors report nearly identical mean errors (≈1.2e‑6) and standard deviations for both implementations, overlapping histograms, and no significant difference in the Kolmogorov‑Smirnov test (p > 0.5). This demonstrates that the GPU kernel matches the CPU baseline in accuracy while delivering the expected speedup.

The paper’s contributions are threefold: (1) a clear statistical framework that separates precision‑induced error from implementation bugs; (2) a practical algorithm and set of analysis tools that can be integrated into CI pipelines for mixed‑precision code; (3) guidance on input generation, oracle selection, and error‑metric choice tailored to various precision levels. Limitations include the computational overhead of high‑precision oracles for large‑scale workloads and the dependence of test coverage on the quality of the input generator. Future work suggests adaptive sampling strategies, oracle acceleration (e.g., using GPU‑based double‑precision for FP16 tests), and extending the methodology to multi‑precision hierarchies.

Overall, Dual‑Delta Testing offers a rigorous, statistically sound, and extensible approach for developers and researchers to validate the numerical fidelity of mixed‑precision algorithms, moving beyond the ambiguous single‑delta paradigm toward reproducible, quantifiable confidence in computational results.