Grammatical Error Correction Evaluation by Optimally Transporting Edit Representation

Automatic evaluation in grammatical error correction (GEC) is crucial for selecting the best-performing systems. Currently, reference-based metrics are a popular choice, which basically measure the similarity between hypothesis and reference sentences. However, similarity measures based on embeddings, such as BERTScore, are often ineffective, since many words in the source sentences remain unchanged in both the hypothesis and the reference. This study focuses on edits specifically designed for GEC, i.e., ERRANT, and computes similarity measured over the edits from the source sentence. To this end, we propose edit vector, a representation for an edit, and introduce a new metric, UOT-ERRANT, which transports these edit vectors from hypothesis to reference using unbalanced optimal transport. Experiments with SEEDA meta-evaluation show that UOT-ERRANT improves evaluation performance, particularly in the +Fluency domain where many edits occur. Moreover, our method is highly interpretable because the transport plan can be interpreted as a soft edit alignment, making UOT-ERRANT a useful metric for both system ranking and analyzing GEC systems. Our code is available from https://github.com/gotutiyan/uot-errant.

💡 Research Summary

The paper addresses a critical limitation of current automatic evaluation methods for Grammatical Error Correction (GEC). Traditional reference‑based metrics, including embedding‑based scores such as BERTScore, primarily measure similarity between the hypothesis and reference sentences at the token level. Because most tokens remain unchanged during correction, these metrics are dominated by unchanged words and fail to capture the quality of the actual edits. To overcome this, the authors propose to evaluate GEC systems by focusing directly on the edits themselves, using the ERRANT framework to extract fine‑grained edit operations from both the hypothesis and the reference.

A novel representation called the “edit vector” is introduced. For each edit e in a set E extracted from a source sentence S, the edit vector V(e, E, S) is defined as the difference between the sentence embedding of the fully corrected sentence S_E and the embedding of the same sentence with edit e removed (S_E \ {e}). The embedding function Enc(·) can be any contextual encoder such as BERT mean‑pooled representations. This construction captures both the direction and magnitude of the semantic shift induced by the edit: the vector direction encodes the type of change, while its ℓ2 norm reflects how much the meaning is altered. Consequently, edits that produce similar semantic effects are mapped to nearby vectors, and more impactful edits obtain larger norms.

Having turned each edit into a high‑dimensional vector, the authors treat the collections of hypothesis edit vectors V_hyp and reference edit vectors V_ref as two discrete distributions. The mass of each point is set to the norm of its edit vector, providing an implicit weighting that emphasizes more consequential edits. The cost matrix C is defined as the Euclidean distance between every pair of hypothesis and reference edit vectors. To compare the two distributions, the paper employs Unbalanced Optimal Transport (UOT), a relaxation of the classic balanced OT formulation. UOT introduces entropy regularization and KL‑divergence penalties that allow portions of mass to remain untransported, which is essential for GEC because systems often over‑correct (producing extra edits) or under‑correct (missing edits). The optimal transport plan T is computed efficiently via the Sinkhorn algorithm.

The transport plan T can be interpreted as a soft alignment between hypothesis and reference edits. The authors decompose T into three scalar scores: True Positive (TP) – the total transported mass, False Positive (FP) – the mass of hypothesis edits that could not be matched, and False Negative (FN) – the mass of reference edits left unmatched. Using these, they compute precision = TP/(TP+FP), recall = TP/(TP+FN), and the Fβ score with β = 0.5, which is standard in GEC evaluation. When multiple references are available, the reference yielding the highest F0.5 is selected, mirroring common practice.

The method, named UOT‑ERRANT, is evaluated on two meta‑evaluation benchmarks: SEED‑A and GMEG‑Data. SEED‑A contains outputs from 14 systems, including recent large language models such as GPT‑3.5, together with human ranking judgments. It distinguishes a “Base” setting and a “Fluency” setting, the latter featuring many edits per sentence. UOT‑ERRANT achieves higher Pearson and Spearman correlations with human rankings than baseline metrics including the original ERRANT, PT‑ERRANT (which uses BERTScore as a scalar edit weight), and pure BERTScore. The improvement is especially pronounced in the Fluency domain, where the metric gains over 10 % in correlation relative to the best prior method.

Beyond correlation, the authors analyze the properties of edit vectors. They find that edits involving content words (nouns, verbs) tend to have larger norms, indicating greater semantic impact, and that vectors cluster by error type, confirming that the representation captures meaningful linguistic distinctions. The transport plan itself provides an interpretable soft alignment, allowing researchers to visualize which hypothesis edits align with which reference edits and to diagnose systematic over‑ or under‑corrections.

In summary, the paper makes three key contributions: (1) a principled way to embed edits as vectors that reflect their semantic effect, (2) the application of unbalanced optimal transport to compare sets of edits, handling the inherent imbalance of GEC outputs, and (3) a new evaluation metric, UOT‑ERRANT, that outperforms existing reference‑based metrics in correlation with human judgments while offering transparent, edit‑level diagnostics. This work advances the state of automatic GEC evaluation by shifting the focus from surface similarity to meaningful edit semantics, and it opens avenues for further research on transport‑based evaluation in other text‑generation tasks.