Optimal smoothing parameter in Eilers-Whittaker smoother

The Eilers-Whittaker method for data smoothing effectiveness depends on the choice of the regularisation parameter, and automatic selection is a necessity for large datasets. Common methods, such as leave-one-out cross-validation, can perform poorly when serially correlated noise is present. We propose a novel procedure for selecting the control parameter based on the spectral entropy of the residuals. We define an S-curve from the Euclidean distance between points in a plot of the spectral entropy of the residuals versus that of the smoothed signal. The regularisation parameter corresponding to the absolute maximum of this S-curve is chosen as the optimal parameter. Using simulated data, we benchmarked our method against cross-validation and the V-curve. Validation was also performed on diverse experimental data. This robust and straightforward procedure can be a valuable addition to the available selection methods for the Eilers smoother.

💡 Research Summary

The paper addresses the critical problem of automatically selecting the regularisation parameter λ in the Eilers‑Whittaker smoothing algorithm, a task that becomes increasingly important as data sets grow larger. Traditional approaches such as leave‑one‑out cross‑validation (LOOCV), the L‑curve, and the V‑curve have been widely used, but they suffer from systematic biases when the noise in the data exhibits serial correlation—a common situation in time‑series measurements. LOOCV tends to underestimate λ under correlated noise, leading to under‑smoothing, while the V‑curve, though more robust, still overestimates λ, especially at higher noise levels.

To overcome these limitations, the authors propose a novel selection criterion based on spectral entropy. For any candidate λ, the residuals R(λ) = y – ŝ(λ) and the differenced smoothed signal Dŝ(λ) are transformed to the frequency domain via the Fourier transform. After normalising the power spectrum, the Shannon entropy H_S = –∑ P_F(q) log P_F(q) is computed for both the residuals and the differenced signal. The pair (log H_R, log H_ŝ) defines a point h_λ in a two‑dimensional entropy space. By calculating the Euclidean distance between successive points, e_λ = ||h_λ+1 – h_λ||, an “S‑curve” is obtained as a function of λ. The absolute maximum of this S‑curve is taken as the optimal λ (denoted λ_o).

The methodology is evaluated through extensive simulations and real‑world experiments. Simulated test functions include a simple sine wave, a composite sinusoidal‑logarithmic function, and more complex analytical expressions. Gaussian white noise and coloured (1/f) noise are added at varying signal‑to‑noise ratios (SNR). For each λ, the mean‑square error (MSE) between the true underlying signal and the smoothed estimate is computed; the λ that minimizes MSE defines the ground‑truth optimal λ_o. Across all simulated scenarios, the spectral‑entropy‑based S‑curve selects λ values that lie almost exactly on the MSE‑optimal line, delivering MSEs comparable to the theoretical optimum. In contrast, LOOCV consistently chooses λ that are too small, while the V‑curve selects λ that are too large, with performance degrading as noise correlation increases.

Real data tests involve three distinct domains: (a) optical spectroscopy where periodic instrumental noise introduces correlation, (b) electrophysiological recordings (EEG) characterized by 1/f background noise, and (c) environmental temperature time series with seasonal trends and autocorrelated residuals. In each case, the S‑curve exhibits a clear, single absolute maximum, and the λ chosen by this maximum yields smoothed curves that visually and quantitatively (via MSE against a high‑quality reference) outperform those obtained with LOOCV or the V‑curve. Notably, under coloured noise conditions the S‑curve maintains stability, whereas the other methods display pronounced bias.

From a computational standpoint, the entropy calculation relies on fast Fourier transforms, giving an O(N log N) cost for a data set of size N. The distance computation across a grid of M λ‑values is O(M), making the overall algorithm scalable to large data sets. The authors demonstrate that a logarithmic λ grid of 50–100 points suffices to locate the absolute maximum reliably, further reducing computational load.

The paper’s contributions are threefold: (1) introduction of a spectral‑entropy‑based objective function that simultaneously captures the information content of residuals and the smoothness of the estimate, (2) formulation of the S‑curve as an intuitive, computationally cheap visual tool for λ selection, and (3) comprehensive validation showing superior performance over established methods across synthetic and experimental data, especially in the presence of serially correlated noise. The authors suggest future extensions to multivariate time series, non‑linear smoothing frameworks, and hybrid schemes that combine the entropy criterion with machine‑learning‑based priors.