Toward Trustworthy Evaluation of Sustainability Rating Methodologies: A Human-AI Collaborative Framework for Benchmark Dataset Construction

Sustainability or ESG rating agencies use company disclosures and external data to produce scores or ratings that assess the environmental, social, and governance performance of a company. However, sustainability ratings across agencies for a single company vary widely, limiting their comparability, credibility, and relevance to decision-making. To harmonize the rating results, we propose adopting a universal human-AI collaboration framework to generate trustworthy benchmark datasets for evaluating sustainability rating methodologies. The framework comprises two complementary parts: STRIDE (Sustainability Trust Rating & Integrity Data Equation) provides principled criteria and a scoring system that guide the construction of firm-level benchmark datasets using large language models (LLMs), and SR-Delta, a discrepancy-analysis procedural framework that surfaces insights for potential adjustments. The framework enables scalable and comparable assessment of sustainability rating methodologies. We call on the broader AI community to adopt AI-powered approaches to strengthen and advance sustainability rating methodologies that support and enforce urgent sustainability agendas.

💡 Research Summary

The paper addresses the pervasive problem of inconsistency among ESG (Environmental, Social, Governance) ratings issued by different agencies, which undermines comparability, credibility, and practical decision‑making for investors, regulators, and corporations. Empirical studies show that correlations between ESG scores range only from 0.38 to 0.71, reflecting divergent scopes, weightings, and measurement approaches. To remedy this, the authors propose a universal human‑AI collaborative framework that creates trustworthy benchmark datasets for evaluating ESG rating methodologies. The framework consists of two complementary components: STRIDE (Sustainability Trust Rating & Integrity Data Equation) and SR‑Delta (Sustainability Rating Discrepancy Analysis).

STRIDE extends the classic “trust equation” (credibility, reliability, intimacy, and self‑orientation) into a formal Human‑Machine Trust Equation tailored for sustainability data. Trust τ(x) is modeled as a sigmoid of a weighted linear combination of four latent components: credibility C(x), reliability R(x), intimacy I(x), and a penalty for self‑serving purpose S(x). Each component is decomposed into measurable sub‑dimensions:

• Credibility (C) includes inclusiveness & materiality (IM), auditability & traceability (AT), exemplary reference (ER), and time relevance (TR). These capture data coverage across geographies, industries, standard layers, external signals (e.g., news sentiment), provenance, high‑quality reference firms, and recency.
• Reliability (R) covers rigorous sampling methodology (SM), ground‑truth annotation (GT), agility with appropriate trade‑offs (AG), and security & safety (SS). This ensures that the modeling pipeline is statistically sound, annotated by experts, balanced between performance and interpretability, and protected against adversarial threats.
• Intimacy (I) reflects the quality of human‑AI interaction through human guidance (HG), explainability (DE), and interface friendliness (IF).
• Self‑Serving Purpose (S) penalizes bias toward agency or stakeholder self‑interest via transparency (T) and stakeholder bias (RS).

The authors provide explicit functional forms for each sub‑dimension, allowing practitioners to compute a numeric trust score for any candidate benchmark dataset. High trust scores indicate that the dataset reflects industry best practices, is auditable, up‑to‑date, and suitable for large‑scale evaluation of ESG rating methods.

SR‑Delta leverages the STRIDE‑validated benchmark to conduct discrepancy analysis across rating agencies. For a given firm, the framework compares each agency’s rating with the benchmark, quantifies the delta, and attributes the source of divergence to specific credibility or reliability dimensions. For instance, an agency that over‑weights a particular environmental metric may be flagged for insufficient inclusiveness (IM) or biased sampling (SM). The resulting diagnostics generate actionable feedback for agencies to refine metric selection, weighting schemes, and data update cycles.

The paper also outlines three systemic challenges in ESG data: (1) heterogeneous disclosure standards leading to inconsistent inputs, (2) limited high‑quality data due to voluntary reporting and fragmented data infrastructures, and (3) the absence of an end‑to‑end trustworthy document acquisition pipeline (retrieval, annotation, provenance tracking, and drift handling). STRIDE addresses these by mandating multi‑layer coverage, audit trails, and temporal decay functions, while SR‑Delta provides a systematic way to monitor and correct methodological drift.

A pilot case study demonstrates the workflow: large language models automatically extract key performance indicators from corporate sustainability reports; human experts validate a subset, ensuring annotation quality. The resulting dataset receives a STRIDE trust score, and discrepancy analysis reveals that 65 % of rating differences between MSCI and Sustainalytics stem from gaps in inclusiveness and sampling. Agencies can then adjust their methodologies accordingly.

Future research directions include dynamic updating mechanisms that ingest real‑time news and social media, multimodal LLMs that process tables, figures, and images, and the development of governance standards for the entire benchmark lifecycle.

In summary, the authors present a rigorous, quantifiable trust framework and a discrepancy‑analysis tool that together enable scalable, comparable, and transparent evaluation of ESG rating methodologies. By integrating human expertise with advanced AI capabilities, the proposed approach promises to enhance the credibility of sustainability assessments and support more informed, responsible decision‑making across the financial ecosystem.