Evaluating Organizational Effectiveness: A New Strategy to Leverage Multisite Randomized Trials for Valid Assessment

Determining which organizations are more effective in implementing an intervention program is essential for theoretically and empirically characterizing exemplary practice and for intervening to enhance the capacity of ineffective ones. Yet sites differ in their local ecological conditions including client composition, alternative programs, and community context. Applying the causal inference framework, this study proposes a formal mathematical definition for the local relative effectiveness of an organization attributable solely to malleable organizational practice. Capitalizing on multisite randomized trials, the identification leverages observed control group outcomes that capture some of the confounding impacts of otherwise unmeasured contextual variation. We propose a two-step mixed-effects modeling (2SME) procedure that adjusts for pre-existing between-site variation. A series of Monte Carlo simulations reveals its superior performance in comparison with conventional methods. We apply the new strategy to an evaluation of Job Corps centers nationwide serving disadvantaged youths.

💡 Research Summary

The paper tackles a fundamental problem in evaluating the effectiveness of organizations that deliver interventions across multiple sites: conventional site‑specific intent‑to‑treat (ITT) estimates conflate the organization’s own contribution with a host of contextual factors such as client composition, availability of alternative services, and local economic conditions. Drawing on causal‑inference terminology, the authors distinguish between a “global relative effectiveness” (GRE) – a hypothetical comparison of all organizations under a synthetic average site – and a “local relative effectiveness” (LRE), which compares organizations only with peers operating under similar ecological conditions. The authors argue that LRE, rather than GRE, is the metric of interest for policymakers who need to know how well an organization performs given its real‑world context.

To isolate LRE, the authors propose a two‑step mixed‑effects (2SME) modeling strategy that exploits the fact that, in a multisite randomized trial, the observed outcomes of the control group at each site capture unmeasured site‑level confounders (e.g., prior vulnerability, community resources). In Step 1, individual‑level pre‑treatment covariates and the site‑specific control‑group mean are entered as fixed and random effects to estimate each site’s baseline risk (the “predisposition‑by‑environment” component). In Step 2, treatment‑group outcomes are modeled conditional on the Step 1 baseline, with a random slope representing the site‑specific organizational effect. This hierarchy explicitly models between‑site heterogeneity, ensuring that the estimated organizational effect reflects performance under comparable local conditions.

Monte‑Carlo simulations compare three estimators: (i) a naïve ITT‑only estimator, (ii) an estimator that adjusts only for the control‑group mean, and (iii) the proposed 2SME. Across a range of scenarios with varying degrees of unmeasured site heterogeneity, 2SME delivers near‑zero bias and the lowest mean‑squared error. When unmeasured confounding is strong, 2SME improves accuracy by more than 30 % relative to the alternatives, demonstrating its robustness.

The methodology is applied to the National Job Corps Study (NJCS), a multisite randomized trial that enrolled all U.S. Job Corps centers in the mid‑1990s. Prior analyses reported an average ITT earnings gain of $1,415 (SE = $358) and a between‑site standard deviation of $1,687, but these figures blend organizational practice with local context. Using 2SME, the authors estimate LRE for each center, effectively “level‑setting” the control‑group outcomes to control for local ecological conditions. Results show that rural centers, whose control‑group earnings were low, exhibit relatively high LRE, whereas urban centers in high‑unemployment areas show lower LRE despite similar ITT gains. This nuanced picture would be invisible to a simple ITT analysis and provides a more equitable basis for contract renewals, performance‑based funding, and targeted capacity‑building.

The paper also discusses limitations. The control‑group mean is an imperfect proxy for all unmeasured contextual factors, especially when those factors evolve over time. The random‑slope assumption imposes linearity on organizational effects, potentially missing nonlinear dynamics. Finally, multisite randomization does not fully eliminate cross‑site contamination or spillover. The authors suggest extensions such as incorporating longitudinal control data, nonlinear mixed‑effects specifications, and network‑aware models to further refine LRE estimation.

In sum, the study offers a principled, empirically validated framework for isolating the pure contribution of organizational practice in multisite randomized trials. By leveraging control‑group outcomes and a two‑step mixed‑effects approach, it delivers less biased, more precise estimates of local relative effectiveness than traditional ITT‑based methods. This advance has immediate relevance for policymakers, funders, and managers seeking fair, context‑sensitive performance metrics for programs such as Job Corps.