Differential Test Functioning via Robust Scaling

In the item response theory (IRT) literature, differential test functioning (DTF) has been conceptualized in terms of how the test response function differs over groups of respondents. This paper presents an alternative approach to DTF that focusses on how the distribution of the latent trait differs over groups, which is referred to as impact. It is proposed to evaluate DTF by comparing two estimates of impact, one that naively aggregates over all test items and a robust alternative that down-weights items that exhibit differential item functioning (DIF). Taking this approach, this paper makes the following three contributions. First it is shown that the difference between the naive and robust estimands provides a convenient effect size for quantifying the extent to which DIF affects conclusions about impact (as opposed to test scores). Second it is shown how to construct a robust estimator that yields consistent estimates of impact whenever fewer than 1/2 of items exhibit DIF. Third, a relatively general purpose Wald test of the difference between two estimates of impact is developed. Using simulations and an empirical example from physics education, it is shown how the proposed effect size and test statistic perform using the proposed robust estimator of impact, as well as estimators that arise from conventional item-by-item tests of DIF.

💡 Research Summary

The paper re‑examines differential test functioning (DTF) from the perspective of impact—the difference in the latent trait distributions between groups—rather than the traditional focus on differences in test scores. Impact is quantified as the standardized mean difference δ₀ = (μ₁ − μ₂)/σ, where σ is the pooled standard deviation of the latent trait. Two estimators of δ₀ are introduced: a naïve estimator δ_U that simply averages item‑level treatment effects across all items, and a robust estimator δ_R that down‑weights items suspected of differential item functioning (DIF). The difference Δ = δ_U − δ_R serves as an effect size measuring how much DIF contaminates conclusions about impact.

Methodologically, the authors work within the two‑parameter logistic (2PL) IRT model. For each item i, a treatment effect δ_i(ν) = (b_{i1} − b_{i2})/ā_i is defined, where ā_i is the average discrimination across groups. When an item exhibits no DIF, δ_i equals the true impact δ₀. Aggregating these δ_i with weights w_i yields a test‑level scaling function δ(ν) = Σ w_i δ_i(ν). The naïve approach uses equal weights (w_i = 1/m). To obtain a robust version, the paper adopts a redescending ψ‑function (specifically Tukey’s bi‑square) to construct weights w_{Ri} = ψ(u_i)/u_i normalized, where u_i = δ_i(ν) − δ is the deviation of each item’s effect from a provisional overall estimate δ. An iteratively re‑weighted least squares (IRLS) algorithm updates δ and the weights until convergence.

A key theoretical contribution is the proof that δ_R is a consistent estimator of δ₀ provided fewer than half of the items exhibit DIF. This aligns with the maximal breakdown point of translation‑equivariant estimators (50 %). Consequently, Δ = 0 can be interpreted as “no DIF impact” only when the robust estimator truly recovers δ₀; otherwise Δ = 0 could arise from pathological cancellation of DIF across items.

The authors develop a Wald test for the null hypothesis Δ = 0. The asymptotic variance of Δ̂ is derived from the covariance matrix of the MLE item parameters and the Jacobian of the ψ‑function. Under large samples, the Wald statistic follows a standard normal distribution, allowing straightforward hypothesis testing.

Simulation studies vary the proportion of DIF items (0 %–50 %), sample sizes (n = 500, 1000), and test lengths (m = 20, 40). Results show that the robust estimator remains unbiased and efficient up to about 30 % DIF, while the naïve estimator becomes increasingly biased. The Wald test maintains the nominal Type I error rate and achieves good power once DIF exceeds roughly 10 % of items. By contrast, traditional anchor‑item DIF procedures deteriorate markedly when more than 25 % of items contain DIF in the same direction.

An empirical illustration uses a physics‑education assessment administered to two instructional groups. The naïve impact estimate is δ_U = 0.48 SD, whereas the robust estimate is δ_R = 0.36 SD, yielding Δ̂ = 0.12 SD (p = 0.018). This indicates that ignoring DIF would overstate the group difference by about one‑tenth of a standard deviation. The analysis demonstrates that the robust method can flag substantive DIF influence without conducting a full item‑by‑item DIF analysis.

The authors provide an R package, robustDIF, which implements the robust scaling, Δ calculation, and Wald test, facilitating adoption by practitioners. They discuss extensions to multidimensional IRT, more than two groups, and non‑normal latent traits, suggesting that the framework is broadly applicable.

In sum, the paper offers a novel, impact‑oriented definition of DTF, introduces a robust scaling estimator with provable consistency under up to 50 % DIF contamination, and supplies a practical statistical test for DIF‑induced bias in impact estimates. This advances both the theory and practice of fair measurement in educational and psychological testing.