Measurement for Opaque Systems: Multi-source Triangulation with Interpretable Machine Learning

We propose a measurement framework for difficult-to-access contexts that uses indirect data traces, interpretable machine-learning models, and theory-guided triangulation to fill inaccessible measurement spaces. Many high-stakes systems of scientific and policy interest are difficult, if not impossible, to reach directly: dynamics of interest are unobservable, data are indirect and fragmented across sources, and ground truth is absent or concealed. In these settings, available data often do not support conventional strategies for analysis, such as statistical inference on a single authoritative data stream or model validation against labeled outcomes. To address this problem, we introduce a general framework for measurement in data regimes characterized by structurally missing or adversarial data. We propose combining multi-source triangulation with interpretable machine learning models. Rather than relying on accuracy against unobservable, unattainable ideal data, our framework seeks consistency across separate, partially informative models. This allows users to draw defensible conclusions about the state of the world based on cross-signal consistency or divergence from an expected state. Our framework provides an analytical workflow tailored to quantitative characterization in the absence of data sufficient for conventional statistical or causal inference. We demonstrate our approach and explicitly surface inferential limits through an empirical analysis of organizational growth and internal pressure dynamics in a clandestine militant organization, drawing on multiple observational signals that individually provide incomplete and biased views of the underlying process. The results show how triangulated, interpretable ML can recover substantively meaningful variation.

💡 Research Summary

The paper introduces a novel measurement framework designed for “opaque” systems—contexts where the phenomena of interest cannot be observed directly, data are fragmented across multiple sources, and ground‑truth information is deliberately concealed or unavailable. Traditional statistical inference and causal‑identification methods rely on a single authoritative data stream or labeled outcomes, assumptions that break down under structurally missing or adversarial data regimes. To overcome these limitations, the authors propose a triangulation approach that combines several partially independent observational traces with interpretable machine‑learning (ML) models, and uses theory‑driven expectations to evaluate model performance not by accuracy but by consistency with those expectations.

The workflow consists of five steps: (1) articulate known macro‑level dynamics from existing assessments; (2) derive specific, falsifiable predictions from theory about how unobserved micro‑processes should manifest in observable behavior; (3) identify observable empirical traces (e.g., event logs, propaganda texts, external media coverage) that align with those predictions; (4) train separate interpretable ML models on each trace, preserving transparency and allowing variable‑importance diagnostics; and (5) assess model outputs against counterfactual expectations, looking for convergence (support for the theory) or divergence (evidence of alternative mechanisms or strategic ambiguity). Errors are treated as informative signals of theory‑data mismatch rather than mere noise.

The framework is demonstrated on a hard‑to‑measure case: Al‑Qaeda in the Arabian Peninsula (AQAP). Macro‑level evidence (membership estimates from REVMOD and conflict dyads from UCDP) shows rapid growth from a few hundred members in 2009 to several thousand by 2016, with a brief territorial gain in Abyan Governorate in 2011. The authors adopt a “growth‑trap” theory, which predicts that rapid expansion outpaces socialization, leading to internal coherence erosion and a shift toward locally‑focused goals. This theory yields three observable mechanisms: (1) recruitment‑socialization imbalances, (2) exit pressures prompting accommodation of new preferences, and (3) delegation of operational discretion to mid‑level leaders.

Corresponding data sources are (a) event data on target selection, (b) topic‑model analysis of AQAP’s propaganda texts, and (c) media sentiment about the group. For each source, interpretable models (decision trees, SHAP‑augmented logistic regressions) are trained. The results show that during the rapid‑growth period the group’s attacks increasingly target domestic Yemeni interests, the textual topics shift toward local grievances, and external coverage frames AQAP as a regional insurgent rather than a trans‑national jihadist. These patterns converge with the growth‑trap predictions, suggesting bottom‑up accommodation. Conversely, divergence from central al‑Qaeda strategic warnings indicates that the observed shifts are not top‑down directives.

Key contributions include: (i) a measurement paradigm that replaces accuracy‑centric validation with theory‑model consistency, (ii) a systematic way to treat model errors as diagnostic evidence, and (iii) a reusable design that can be ported to other adversarial domains such as illicit networks, supply‑chain risk, or policy analysis. The authors acknowledge limitations: the approach requires at least two independent data streams, depends on the quality of the underlying theory, and involves expert judgment in model specification and interpretation. Nonetheless, the framework offers a pragmatic solution for analysts and policymakers who must draw defensible conclusions from fragmented, opaque data environments where conventional inferential tools are structurally infeasible. Future work is suggested on automating trace selection, extending to Bayesian triangulation, and applying the method to other high‑stakes domains.