AI-assisted German Employment Contract Review: A Benchmark Dataset

Employment contracts are used to agree upon the working conditions between employers and employees all over the world. Understanding and reviewing contracts for void or unfair clauses requires extensive knowledge of the legal system and terminology. Recent advances in Natural Language Processing (NLP) hold promise for assisting in these reviews. However, applying NLP techniques on legal text is particularly difficult due to the scarcity of expert-annotated datasets. To address this issue and as a starting point for our effort in assisting lawyers with contract reviews using NLP, we release an anonymized and annotated benchmark dataset for legality and fairness review of German employment contract clauses, alongside with baseline model evaluations.

💡 Research Summary

This paper introduces “AI-assisted German Employment Contract Review: A Benchmark Dataset,” a pioneering resource designed to advance Natural Language Processing (NLP) applications in legal document analysis, specifically for German-language employment contracts. The core contribution is a curated, expert-annotated dataset, released under a CC BY-NC 4.0 license, alongside comprehensive baseline evaluations of various AI models.

The research addresses a significant gap in Legal NLP: the scarcity of high-quality, domain-specific datasets for languages other than English. Employment contracts are critical legal documents, but reviewing them for void or unfair clauses requires deep legal expertise, making the process costly and time-consuming. While NLP holds promise for automation, the lack of annotated data has hindered progress.

The dataset was created in collaboration with a German law firm specializing in economic law. It contains 1,094 anonymized clauses extracted from real employment contracts. Each clause is annotated with three key pieces of information: a legality label (‘valid’, ‘unfair’, or ‘void’), one of 14 predefined categories (e.g., Compensation, Termination, Garnishment), and, for problematic clauses, a short legal explanation. The annotations were performed by two qualified lawyers through an iterative three-round process. After establishing a common gold standard, an impressive inter-annotator agreement of 96.4% was achieved for the legality labels. The data distribution reveals notable imbalances; for instance, clauses in categories like “Garnishment/Assignment” have a 67.9% rate of being void/unfair, largely due to specific changes in German law, whereas the “Other” category has only a 9.6% rate. This reflects how legal amendments disproportionately affect certain contract areas.

In the second major part of the paper, the authors establish performance baselines by evaluating several state-of-the-art language models on the binary classification task of identifying “problematic” (void or unfair) vs. “okay” (valid) clauses. They experiment with both open-source (e.g., Bert-Based-German-Cased) and closed-source models (OpenAI’s Ada, GPT-3.5, and GPT-4), employing two primary techniques: prompt engineering and fine-tuning. The input to the models was varied to study the impact of additional context, including using the clause text alone, concatenating the clause with its section title, providing a German-language system prompt casting the AI as a specialized lawyer, and combinations thereof.

The results provide valuable insights. The best overall performance, measured by the F1-score for the problematic class, was achieved by the GPT-3.5-turbo model fine-tuned with the German system instruction prompt. The experiments also demonstrated that providing the section title of the clause generally improved model performance, highlighting the importance of structural context in legal text analysis. Furthermore, the open-source Bert-Based-German-Cased model delivered respectable results, indicating a viable path for research and application without relying on proprietary APIs.

The paper acknowledges potential biases in the dataset, as the source contracts came from clients seeking legal review, which might skew towards more problematic documents. However, the authors argue this reflects a real-world application scenario. By publicly releasing this benchmark dataset and detailed baseline results, the research provides a crucial foundation for the community to build upon, fostering the development of more accurate and efficient AI tools for assisting lawyers and enhancing accessibility in legal contract review.