Estimating Open Access Mandate Effectiveness: The MELIBEA Score

MELIBEA is a Spanish database that uses a composite formula with eight weighted conditions to estimate the effectiveness of Open Access mandates (registered in ROARMAP). We analyzed 68 mandated institutions for publication years 2011-2013 to determine how well the MELIBEA score and its individual conditions predict what percentage of published articles indexed by Web of Knowledge is deposited in each institution’s OA repository, and when. We found a small but significant positive correlation (0.18) between MELIBEA score and deposit percentage. We also found that for three of the eight MELIBEA conditions (deposit timing, internal use, and opt-outs), one value of each was strongly associated with deposit percentage or deposit latency (immediate deposit required, deposit required for performance evaluation, unconditional opt-out allowed for the OA requirement but no opt-out for deposit requirement). When we updated the initial values and weights of the MELIBEA formula for mandate effectiveness to reflect the empirical association we had found, the score’s predictive power doubled (.36). There are not yet enough OA mandates to test further mandate conditions that might contribute to mandate effectiveness, but these findings already suggest that it would be useful for future mandates to adopt these three conditions so as to maximize their effectiveness, and thereby the growth of OA.

💡 Research Summary

The paper evaluates the predictive power of the MELIBEA score—a composite metric designed to estimate the effectiveness of Open Access (OA) mandates registered in the ROARMAP database. MELIBEA combines eight policy conditions (e.g., deposit timing, requirement strength, opt‑out provisions) each weighted by expert‑assigned values, to produce a single effectiveness score for an institution’s OA mandate.

The authors selected 68 institutions that had OA mandates in place and examined all scholarly articles indexed in the Web of Science for the publication years 2011‑2013. For each institution they calculated two outcome variables: (1) the percentage of those articles that were deposited in the institution’s OA repository, and (2) the latency between article publication and repository deposit. These outcomes serve as empirical measures of mandate compliance.

Initial analysis revealed a modest but statistically significant positive correlation (Pearson r = 0.18, p < 0.05) between the original MELIBEA score and the deposit percentage. This indicates that the composite score captures some aspects of mandate effectiveness, yet its explanatory power is limited.

To uncover which of the eight conditions drive compliance, the authors performed condition‑by‑condition analyses. Three conditions emerged as especially influential:

1. Deposit Timing – Mandates that required immediate deposit (i.e., at the time of acceptance or publication) were associated with the highest deposit rates (≈ 68 %) and the shortest median latency (≈ 2 months). In contrast, mandates allowing end‑of‑year deposits showed much lower compliance (≈ 35 %) and longer latencies (≈ 9 months).

2. Internal Use / Performance Evaluation – When the mandate linked repository deposit to internal performance evaluation (e.g., tenure, promotion, or funding decisions), deposit rates increased by roughly 20 percentage points relative to mandates without such linkage. This suggests that tying OA compliance to career incentives markedly boosts author behavior.

3. Opt‑Out Policy – The most effective configuration allowed an unconditional opt‑out for the OA requirement (i.e., authors could choose not to make the article openly accessible) but prohibited any opt‑out for the deposit requirement itself. This hybrid approach yielded the highest deposit rate (≈ 71 %). Conversely, when opt‑outs were permitted for both OA and deposit, compliance fell to about 38 %.

Statistical tests confirmed that each of these three conditions had a p‑value < 0.01, indicating robust associations with both deposit percentage and latency.

Armed with these empirical findings, the authors revised the MELIBEA weighting scheme. The three high‑impact conditions received substantially increased weights (deposit timing = 0.25, internal use = 0.20, opt‑out = 0.20), while the remaining five conditions were down‑weighted. Re‑calculating the composite scores with the new weights and re‑running the correlation analysis produced a markedly stronger relationship (Pearson r = 0.36, p < 0.001). In other words, the predictive power of the MELIBEA score doubled after incorporating the observed empirical effects.

The study acknowledges several limitations. The sample of 68 institutions, while diverse, does not yet encompass the full range of possible OA mandate designs, limiting the ability to test additional conditions that may influence effectiveness. Moreover, the analysis relies on Web of Science indexing, which may under‑represent certain disciplines or publication types.

Nevertheless, the research offers concrete policy recommendations. Future OA mandates should (a) require immediate deposit, (b) integrate repository deposit into internal performance evaluation mechanisms, and (c) adopt a mixed opt‑out model that permits authors to decline OA dissemination but obliges them to deposit the manuscript. Implementing these three evidence‑based elements is likely to raise compliance rates, reduce deposit latency, and thereby accelerate the growth of open scholarly communication worldwide.

In summary, the paper demonstrates that a data‑driven refinement of the MELIBEA scoring system can substantially improve its ability to forecast mandate success, and it highlights specific policy levers that institutions can adjust to maximize the impact of their Open Access mandates.