The Need for Standardized Evidence Sampling in CMMC Assessments: A Survey-Based Analysis of Assessor Practices

The Cybersecurity Maturity Model Certification (CMMC) framework provides a common standard for protecting sensitive unclassified information in defense contracting. While CMMC defines assessment objectives and control requirements, limited formal guidance exists regarding evidence sampling, the process by which assessors select, review, and validate artifacts to substantiate compliance. Analyzing data collected through an anonymous survey of CMMC-certified assessors and lead assessors, this exploratory study investigates whether inconsistencies in evidence sampling practices exist within the CMMC assessment ecosystem and evaluates the need for a risk-informed standardized sampling methodology. Across 17 usable survey responses, results indicate that evidence sampling practices are predominantly driven by assessor judgment, perceived risk, and environmental complexity rather than formalized standards, with formal statistical sampling models rarely referenced. Participants frequently reported inconsistencies across assessments and expressed broad support for the development of standardized guidance, while generally opposing rigid percentage-based requirements. The findings support the conclusion that the absence of a uniform evidence sampling framework introduces variability that may affect assessment reliability and confidence in certification outcomes. Recommendations are provided to inform future CMMC assessment methodology development and further empirical research.

💡 Research Summary

The paper investigates the largely undocumented practice of evidence sampling within Cybersecurity Maturity Model Certification (CMMC) assessments, focusing on Level 2 assessments that rely on Certified Third‑Party Assessment Organizations (C3PAOs) and certified assessors. Recognizing that CMMC specifies control requirements and assessment objectives but offers scant guidance on how assessors should select, review, and validate evidence, the authors conducted an exploratory, mixed‑methods survey of 17 experienced CMMC assessors (both Certified CMMC Assessors and Lead Certified CMMC Assessors).

The survey instrument was deliberately crafted to map onto core dimensions of evidence sampling: assessor background, decision drivers, application of sampling in structured scenarios, perceived consistency across assessments, and attitudes toward standardization. It combined closed‑ended items for quantitative comparison with open‑ended prompts to capture nuanced reasoning. Data collection was anonymous, targeting only individuals listed in the official CyberAB marketplace, and was approved by an institutional IRB. After cleaning incomplete responses, the authors performed descriptive profiling, thematic coding of qualitative answers, and integration of findings across data types.

Key findings include:

1. Lack of Formal Guidance – 71 % of respondents reported that their C3PAO does not provide a sampling methodology, and 44 % indicated that no formal guidance exists at all. Only a minority (13 %) described existing guidance as highly detailed.

2. Dominant Decision Factors – When asked what drives sampling scope, respondents most frequently cited environment size/complexity (82 %), assessor experience/judgment (76 %), and risk or control criticality (71 %). Only 12 % reported using a quantitative or statistical model, underscoring the reliance on subjective judgment.

3. Diverse Practices in Controlled Scenarios – In a scenario asking how much of a medium‑sized organization’s user accounts should be sampled for access‑control compliance, answers ranged from 1‑3 % to 95‑100 %, with many participants refusing to give a flat percentage and instead describing stratified or coverage‑based approaches. For a high‑risk control (e.g., incident response), 47 % answered “Depends,” emphasizing contextual representativeness rather than a fixed quota. Similar dispersion appeared for documentation and record‑review sampling.

4. Inconsistency Is Common – 71 % of participants observed sampling inconsistencies between assessors at least occasionally (53 % “occasionally,” 18 % “very frequently”). This inconsistency was reported across experience levels, suggesting a systemic issue rather than a novice problem.

5. Support for Standardization, but Not Rigid Percentages – While a majority expressed strong support for the development of standardized evidence‑sampling guidance, they simultaneously opposed prescriptive percentage‑based requirements, preferring a risk‑informed, flexible framework that still ensures minimum rigor and traceability.

The authors interpret these results through the lens of audit theory, noting that current CMMC practice falls short of principles such as representativeness, consistency, and traceability. They argue that the absence of a uniform, risk‑based sampling framework introduces variability that can undermine confidence in certification outcomes. Consequently, they recommend that C3PAOs and the CMMC governing body develop a risk‑informed sampling methodology—one that defines minimum evidence‑coverage thresholds, encourages stratified or purposive sampling based on control criticality, and documents the rationale for sampling decisions.

Future research directions include expanding the sample size, conducting longitudinal studies of actual assessment engagements, and exploring automated tools or statistical models that could operationalize risk‑based sampling in the CMMC context. By providing empirical evidence of current practice variability, the paper makes a compelling case for formalizing evidence‑sampling guidance to enhance the reliability, fairness, and credibility of CMMC certifications.