A shift-optimized Hill-type estimator

A wide range of natural and social phenomena result in observables whose distributions can be well approximated by a power-law decay. The well-known Hill estimator of the tail exponent provides results which are in many respects superior to other estimators in case the asymptotics of the distribution is indeed a pure power-law, however,systematic errors occur if the distribution is altered by simply shifting it. We demonstrate some related problems which typically emerge when dealing with empirical data and suggest a procedure designed to extend the applicability of the Hill estimator.

💡 Research Summary

**
The paper addresses a well‑known problem in extreme‑value statistics: the Hill estimator, which is the standard tool for estimating the tail exponent α of a power‑law distribution, is highly sensitive to a simple additive shift of the data. In many empirical settings the observed variable Y is related to the underlying power‑law variable X by Y = X + s, where s is a constant offset introduced by measurement, preprocessing, or a genuine location parameter. The classic Hill estimator assumes a pure Pareto tail P(X > x) = C x⁻ᵅ and therefore ignores any shift. When a shift is present, the logarithmic ratios used in the Hill formula are distorted, leading to a systematic bias that grows linearly with s and inversely with the chosen threshold k (the number of top order statistics). Consequently, the estimator can severely over‑ or under‑estimate α, especially when the tail sample is small.

To remedy this, the authors propose the “Shift‑Optimized Hill Estimator” (SOHE). The method consists of two main steps. First, a shift parameter ŝ is estimated by scanning a grid of candidate shifts {s_j}. For each candidate, the data are shifted back (Z_i = X_i − s_j) and the Hill estimator is applied to the top k observations of Z. The quality of the fit is assessed by a distance measure between the empirical tail distribution of Z and the theoretical Pareto model; the authors use the Kolmogorov‑Smirnov (KS) statistic, Anderson‑Darling, or the sum of squared residuals from a log‑log linear regression. The optimal shift ŝ is the one that minimizes this distance, i.e. the shift that makes the tail look most Pareto‑like.

Second, with the estimated shift applied, the ordinary Hill formula is used on the shifted data to obtain the final tail‑exponent estimate α̂_SOHE. The choice of k can be made by any standard technique (visual Hill plot, minimizing mean‑squared error, etc.) because the shift correction does not alter the bias‑variance trade‑off inherent to the Hill estimator. The authors also discuss a joint optimization over (s, k) by defining a composite objective L(s, k) = KS(s, k) + λ·Var