Online platforms require robust systems to enforce content safety policies at scale. A critical component of these systems is the ability to evaluate the quality of moderation decisions made by both human agents and Large Language Models (LLMs). However, this evaluation is challenging due to the inherent trade-offs between cost, scale, and trustworthiness, along with the complexity of evolving policies. To address this, we present a comprehensive Decision Quality Evaluation Framework developed and deployed at Pinterest. The framework is centered on a high-trust Golden Set (GDS) curated by subject matter experts (SMEs), which serves as a ground truth benchmark. We introduce an automated intelligent sampling pipeline that uses propensity scores to efficiently expand dataset coverage. We demonstrate the framework's practical application in several key areas: benchmarking the cost-performance trade-offs of various LLM agents, establishing a rigorous methodology for data-driven prompt optimization, managing complex policy evolution, and ensuring the integrity of policy content prevalence metrics via continuous validation. The framework enables a shift from subjective assessments to a data-driven and quantitative practice for managing content safety systems.
User trust is paramount to the success of online platforms. At Pinterest, our goal is to cultivate a safe and inspiring environment for our users. To achieve this, we establish comprehensive content safety policies and community guidelines across various policy areas [10]. These broad principles are then translated into detailed internal enforcement policies for our operational teams. This complex policy landscape requires scalable enforcement through a combination of machine learning models, human agent labeling, and, increasingly, Large Language Models (LLMs). The integrity of decisions guided by these internal policies directly impacts user safety and trust.
The broader challenge of platform governance at scale is a significant area of research, encompassing the technical and political complexities of algorithmic content moderation [4]. Studies have explored the critical dynamics of human-machine collaboration in these moderation systems [5] and conducted large-scale empirical investigations into how community norms are enforced on major platforms [2]. While this prior work establishes the operational and social challenges of moderation, our paper addresses a complementary and foundational need: the development of a rigorous and scalable evaluation framework to measure and ensure the quality and consistency of those moderation decisions.
The use of Generative AI has become critical in our content safety operations, from prevalence measurement [3] to complex policy interpretation. These powerful systems, however, must operate within a landscape defined by several related challenges. A significant challenge in the Content-Safety domain is the inherent ambiguity and complexity of policies, which leads to inconsistent application by both human and automated agents. This is compounded by the extreme rarity of content and the high cost associated with obtaining high-quality labels from subject matter experts (SMEs). These pressures give rise to a fundamental tradeoff between trustworthiness, scale, and cost, a concept we term the “Pyramid of Truth” shown in Figure1. At the apex are the expensive but essential SMEs, while the base consists of scalable but less reliable agents, creating a persistent bottleneck for creating high-quality evaluation data. Without a systematic way to manage this trade-off, several critical problems emerge: These challenges are central to the well-studied problem of learning from noisy labels, where statistical models infer a consensus ground truth by weighting each labeler’s reliability. Such models can effectively quantify cost-quality trade-offs, for instance, by estimating how many non-expert votes are equivalent to a single expert judgment [13]. However, our Decision Quality Evaluation Framework takes a complementary and operational approach. Rather than treating ground truth as a statistical estimate, we use experts in the highly leveraged role of curating a stable, ground truth GDS. This GDS then serves as an explicit and reliable benchmark to measure the quality of all other agents, from scalable human teams to LLMs.
This framework is especially critical for effectively managing LLMs. A key insight is that it is fundamentally easier to move the needle on LLM decision quality than on non-expert human quality. Unlike a global workforce of human agents, which is costly and slow to retrain, an LLM’s decision-making process can be altered in seconds via prompt engineering. However, harnessing this agility requires a rigorous and reliable evaluation framework to measure the impact of these changes. Without one, “prompt optimization” remains a subjective art rather than a data-driven science.
This paper makes the following contributions:
• We introduce a comprehensive framework for decision quality evaluation, featuring an automated intelligent sampling pipeline that uses propensity scores to efficiently curate and expand dataset coverage. Through this framework, we can continuously validate the trustworthiness of enforcement decisions, set explicit quality standards for different use cases, and ensure the reproducible evaluation of all agents within our content safety ecosystem.
The remainder of the paper is structured as follows: Section 2 details our proposed framework, including the core concepts of the GDS, the metrics used to evaluate decision and dataset quality, and the overall system design. Section 3 presents several practical applications of the framework at Pinterest, demonstrating its utility in areas such as LLM prompt optimization and validating prevalence measurements. Finally, Section 4 offers concluding remarks and discusses the broader implications of our work.
The proposed evaluation framework is built upon the foundational concepts of trustworthiness and representativeness to enable reliable offline evaluation of decision quality.
Trustworthiness ensures that each label within our evaluation datasets accurately reflects the intent of the written policy.
This content is AI-processed based on open access ArXiv data.