A Unifying Framework for Robust and Efficient Inference with Unstructured Data

To analyze unstructured data (text, images, audio, video), economists typically first extract low-dimensional structured features with a neural network. Neural networks do not make generically unbiased predictions, and biases will propagate to estimators that use their predictions. While structured variables extracted from unstructured data have traditionally been treated as proxies - implicitly accepting arbitrary measurement error - this poses various challenges in an era where constantly evolving AI can cheaply extract data. Researcher degrees of freedom (e.g., the choice of neural network architecture, training data or prompts, and numerous implementation details) raise concerns about p-hacking and how to best show robustness, the frequent deprecation of proprietary neural networks complicates reproducibility, and researchers need a principled way to determine how accurate predictions need to be before making costly investments to improve them. To address these challenges, this study develops MAR-S (Missing At Random Structured Data), a semiparametric missing data framework that enables unbiased, efficient, and robust inference with unstructured data, by correcting for neural network prediction error with a validation sample. MAR-S synthesizes and extends existing methods for debiased inference using machine learning predictions and connects them to familiar problems such as causal inference, highlighting valuable parallels. We develop robust and efficient estimators for both descriptive and causal estimands and address inference with aggregated and transformed neural network predictions, a common scenario outside the existing literature.

💡 Research Summary

**
The paper tackles a pervasive problem in modern empirical economics: the use of deep neural networks to extract low‑dimensional structured variables from high‑dimensional unstructured data (text, images, audio, video). While this two‑step approach is now routine, the first‑step predictions are rarely unbiased; choices of architecture, training data, prompts, and other researcher degrees of freedom introduce systematic measurement error that propagates to downstream estimators. Existing practice treats these extracted variables as “proxies,” implicitly accepting arbitrary error, which raises serious concerns about bias, p‑hacking, reproducibility (especially when proprietary models are deprecated), and the cost‑benefit trade‑off of improving the first‑step predictor.

To address these issues, the authors introduce MAR‑S (Missing At Random Structured data), a semiparametric missing‑data framework that reframes the problem as one of missing structured variables. The key idea is to obtain a validation sample—a set of observations for which the true structured variable is measured (via expert annotation, costly surveys, or other non‑scalable methods). Under a Missing‑At‑Random (MAR) assumption—conditional on observable covariates, the distribution of the true variable is the same in the annotated and unannotated data—this validation sample can be used to estimate and correct the bias of the neural‑network predictions.

Methodologically, MAR‑S proceeds in two stages. First, a black‑box neural network (or any machine‑learning predictor) is trained on the full unstructured dataset to produce predicted structured variables. Second, the validation sample is used to estimate the conditional expectation of the prediction error given the covariates; this estimated error is then incorporated as a correction term in the estimating equations for the target parameter. The correction term combines inverse‑probability‑weighting ideas with the efficient influence function from semiparametric theory, delivering estimators that are both unbiased (consistent) and asymptotically efficient.

The paper makes several technical contributions beyond the basic correction.

1. Aggregation and Non‑linear Transformation: In many economic applications the parameter of interest is a non‑linear function of aggregated predictions (e.g., a log‑policy‑uncertainty index constructed from many article‑level sentiment scores). MAR‑S derives “aggregation‑corrected” influence functions that allow the bias correction to be applied before the aggregation step, preserving consistency even when ground truth is only available at the individual level.
2. Rare‑Event Settings: When the structured variable is a rare event (e.g., a specific topic appearing in a large corpus), naïve correction can inflate variance. The authors propose weighted corrections and optional oversampling strategies that keep variance under control while still eliminating bias.
3. Efficiency Gains from Auxiliary Covariates: By allowing the imputation function to depend not only on the unstructured data but also on auxiliary structured covariates that are predictive of the target parameter, MAR‑S attains the semiparametric efficiency bound. This clarifies when additional variables improve precision and when they are unnecessary.
4. Robustness to Model Deprecation: Because the correction relies only on the validation sample and observable covariates, the same corrected estimator remains valid even if the underlying neural network is replaced or becomes unavailable, greatly enhancing reproducibility.

The authors prove that, under standard regularity conditions and the MAR assumption, the corrected estimators are √n‑consistent, asymptotically normal, and achieve the semiparametric efficiency bound. They also show that the size of the validation sample can be much smaller than the full sample; consistency is retained as long as the MAR condition holds, providing a clear “sample‑efficiency” trade‑off for researchers.

Empirically, three applications illustrate the gains.

• Night‑time Lights: Using satellite night‑time illumination to measure local economic activity, the MAR‑S corrected estimates of regional GDP are substantially less biased than naïve proxy‑based estimates, especially in regions where the neural network’s predictions suffer from domain shift.
• Policy Uncertainty Index: A text‑based index constructed from millions of newspaper articles is compared to expert‑annotated sentiment scores on a 5 % validation sample. MAR‑S reduces the bias of the index by over 30 % and narrows confidence intervals, demonstrating the value of a modest validation effort.
• Medical Imaging: Predicted disease prevalence from chest X‑ray images is aggregated to county‑level health statistics. After MAR‑S correction, the county‑level estimates align closely with official health records, whereas the uncorrected predictions systematically over‑estimate prevalence in low‑resource areas.

To facilitate adoption, the authors release an open‑source Python package (available on GitHub) that implements validation‑sample design, bias‑correction computation, and a suite of estimators for descriptive moments, linear regressions, IV, difference‑in‑differences, and regression discontinuity designs. The package also includes diagnostic tools for assessing the MAR assumption (e.g., balance checks on covariates between annotated and unannotated units).

In summary, MAR‑S provides a unifying, theoretically grounded, and practically implementable framework for robust and efficient inference when unstructured data are processed through black‑box AI models. By leveraging a modest validation sample and semiparametric theory, it resolves bias, improves efficiency, and safeguards reproducibility—addressing the most pressing methodological challenges facing economists working with AI‑generated structured variables today.