Effective sampling for large-scale automated writing evaluation systems

Automated writing evaluation (AWE) has been shown to be an effective mechanism for quickly providing feedback to students. It has already seen wide adoption in enterprise-scale applications and is starting to be adopted in large-scale contexts. Training an AWE model has historically required a single batch of several hundred writing examples and human scores for each of them. This requirement limits large-scale adoption of AWE since human-scoring essays is costly. Here we evaluate algorithms for ensuring that AWE models are consistently trained using the most informative essays. Our results show how to minimize training set sizes while maximizing predictive performance, thereby reducing cost without unduly sacrificing accuracy. We conclude with a discussion of how to integrate this approach into large-scale AWE systems.

💡 Research Summary

The paper addresses a critical bottleneck in the deployment of Automated Writing Evaluation (AWE) systems at scale: the high cost of obtaining human‑scored essays for model training. Traditional AWE pipelines require several hundred scored essays per prompt to achieve acceptable predictive performance, which translates into thousands of dollars when the per‑essay scoring cost is $3–$6. The authors propose to dramatically reduce this expense by selecting the most informative essays for human scoring through optimal experimental design and active‑learning techniques, thereby maintaining or even improving model accuracy with far fewer labeled examples.

Data and Feature Extraction
The study uses the eight training/test sets from the 2012 Automated Student Assessment Prize (ASAP) competition, each representing a distinct prompt and scoring rubric. Essays are represented by a 28‑dimensional feature vector derived from the Intelligent Essay Assessor, covering mechanics, grammar, lexical sophistication, and style. The target variable is the integer score (or sum of scores) assigned by human raters.

Regression Model
Because the number of features (p = 28) can exceed the number of training samples (m) in the low‑sample regime, the authors employ ridge regression rather than ordinary least squares. The regularization parameter λ is selected via cross‑validation. Predicted real‑valued scores are mapped back to the discrete rubric using a simple midpoint threshold rule.

Sampling Algorithms
Three algorithms are evaluated for selecting a subset ξ of m essays from the pool of n candidate essays:

1. Fedorov Exchange (D‑optimality) – A greedy exchange procedure that maximizes the determinant of the information matrix M(ξ) = (1/m)XᵀX. Multiple random initializations are run to avoid local optima, and the best design is kept. This method tends to pick points far from the centroid, thereby maximizing the spread of the selected feature vectors.

2. Kennard‑Stone – Starts with the two most distant points in the feature space, then iteratively adds the point whose minimum distance to the current design is maximal. Distances are computed in Mahalanobis space, resulting in a design that is both peripheral and uniformly spread.

3. k‑means Sampling – Performs k‑means clustering with k = m, then selects the actual data point nearest each cluster centroid. This yields an approximately uniform coverage of the space but may miss extreme peripheral points.

The authors also define a persistence metric to quantify how often selections at size m − 1 are retained when the design grows to size m. Kennard‑Stone exhibits perfect persistence, Fedorov shows moderate persistence that improves with larger m, and k‑means has the lowest persistence due to changing cluster centroids.

Experimental Procedure
For each desired training size m (10, 20, … 100), the authors simulate an operational setting by randomly sampling half of the original training set (without replacement) and then applying each sampling algorithm to choose m essays for “human scoring.” Since the true scores are already known, the scoring step is simulated. A ridge regression model is trained on the selected essays, and its performance is measured on the fixed test set using Pearson’s correlation coefficient between rounded predictions and human scores.

Results
All three informed sampling strategies outperform random selection, especially when m is small (30–50 essays). The D‑optimal design (Fedorov) consistently yields the highest correlation, indicating that selecting maximally informative, widely spaced points reduces variance in the estimated regression coefficients. Kennard‑Stone performs nearly as well and has the advantage of deterministic, fully persistent selections, which is valuable for incremental model updates. k‑means, while providing uniform coverage, lags behind in correlation but still surpasses random baselines.

Cost Implications
Assuming a $3–$6 per‑essay scoring cost, reducing the required training set from 500 to roughly 100 essays per prompt cuts the expense from $1,500–$3,000 to $300–$600, a savings of 80 % per prompt. In large‑scale deployments (e.g., MOOCs with hundreds of prompts), this translates into multi‑hundred‑thousand‑dollar reductions.

Integration into Live AWE Systems
The paper outlines a practical workflow for embedding the sampling module into an online AWE platform such as edX. New essays are continuously collected; the system periodically runs the chosen sampling algorithm on the pool of unscored essays, flags the most informative ones for human raters, and retrains the regression model as soon as scores become available. This closed‑loop approach maintains up‑to‑date models while keeping human‑scoring effort minimal.

Conclusions and Future Work
The study demonstrates that optimal‑design‑based sampling can substantially lower the human‑labeling burden in AWE without sacrificing predictive quality. By aligning the assumptions of the sampling algorithm (linear regression) with those of the learning model (ridge regression), the authors achieve robust performance across diverse prompts and scoring ranges. Future research directions include extending the methodology to non‑linear models (e.g., neural networks), handling multi‑trait scoring, and exploring adaptive strategies that jointly optimize sampling and model hyper‑parameters in real time.

Overall, the paper provides a compelling, empirically validated solution to a major scalability challenge in automated writing assessment, bridging statistical experimental design with modern educational technology.