Asymptotic Standard Errors for Reliability Coefficients in Item Response Theory

In a recent review, Liu, Pek, & Maydeu-Olivares (2025b) classified reliability coefficients into two types: classical test theory (CTT) reliability and proportional reduction in mean squared error (PRMSE). This article focuses on quantifying the sampling variability of these coefficients under item response theory (IRT) models. While some existing standard error (SE) formulas are accurate when variability arises only from item parameter estimation, the reliability estimators considered in our work involve additional variability from substituting population moments with sample moments. We propose a general strategy to derive SEs that incorporates both sources of sampling error simultaneously, enabling the estimation of model-based reliability coefficients and their SEs in such settings. We then apply our general theory to derive SEs for two specific estimators under the graded response model: (1) CTT reliability for the expected a posteriori score of the latent variable and (2) PRMSE for the latent variable. Simulation results show that the derived SEs accurately capture the sampling variability across various test lengths in moderate to large samples. We conclude with an empirical illustration and directions for future research.

💡 Research Summary

This paper addresses a gap in the psychometric literature concerning the quantification of sampling variability for reliability coefficients derived under Item Response Theory (IRT) models. While previous work has provided standard error (SE) formulas that are accurate when variability stems solely from the estimation of item parameters, most practical reliability estimators also replace population moments (e.g., variances and covariances implied by the model) with their sample counterparts. This dual source of uncertainty—item‑parameter estimation and sample‑moment substitution—has not been jointly addressed.

The authors propose a general asymptotic framework that simultaneously captures both sources of error. Starting from the maximum‑likelihood estimator (MLE) of the item parameters ν̂, they treat any reliability estimator as a smooth function g(ν, M) of the parameters ν and a vector of sample moments M (e.g., empirical variances of observed scores or of posterior estimates). Using multivariate Taylor expansion and the Delta method, they derive the joint asymptotic distribution of (ν̂ , M̂ ), where M̂ is a sample‑based statistic. The resulting expression is

√n