Removing Manually-Generated Boilerplate from Electronic Texts: Experiments with Project Gutenberg e-Books

Collaborative work on unstructured or semi-structured documents, such as in literature corpora or source code, often involves agreed upon templates containing metadata. These templates are not consistent across users and over time. Rule-based parsing of these templates is expensive to maintain and tends to fail as new documents are added. Statistical techniques based on frequent occurrences have the potential to identify automatically a large fraction of the templates, thus reducing the burden on the programmers. We investigate the case of the Project Gutenberg corpus, where most documents are in ASCII format with preambles and epilogues that are often copied and pasted or manually typed. We show that a statistical approach can solve most cases though some documents require knowledge of English. We also survey various technical solutions that make our approach applicable to large data sets.

💡 Research Summary

The paper addresses a pervasive problem in large‑scale text corpora: the presence of manually inserted boiler‑plate material such as preambles, epilogues, copyright notices, and other template‑like sections. In the Project Gutenberg collection, each e‑book typically begins with a standard header (“*** START OF THIS PROJECT GUTENBERG EBOOK ***”) and ends with a similar footer, but many files also contain author‑written introductions or custom notices that vary from book to book. Traditional rule‑based parsers rely on hand‑crafted regular expressions or ad‑hoc scripts; these are brittle because templates evolve over time, differ across contributors, and require continual maintenance.

The authors propose a statistical, frequency‑driven approach that automatically discovers and removes boiler‑plate without any prior knowledge of the exact templates. Their methodology consists of several stages:

1. Data Acquisition and Pre‑processing – All ASCII e‑books in the Gutenberg archive (≈30,000 titles) are downloaded. For each file, the first and last 200 lines are designated as “regions of interest” where boiler‑plate is most likely to appear.

2. n‑gram Frequency Analysis – All 1‑ to 5‑grams are extracted from the entire corpus and from the regions of interest. Global frequencies are compared with local frequencies; strings that appear disproportionately often in the start or end regions are flagged as candidates.

3. Candidate Selection – A relative frequency threshold (e.g., 80 % higher occurrence in the region than in the whole corpus) is applied. This filters out common English words while retaining repeated template fragments.

4. Clustering and Pattern Extraction – Candidate strings are grouped using a Levenshtein‑distance based similarity matrix and DBSCAN clustering. Each cluster yields a representative pattern that captures variations (e.g., different years, author names) within a single template.

5. Language‑Based Filtering – To avoid false positives from frequently used natural‑language phrases, the system cross‑checks candidates against an English word list and a simple part‑of‑speech tagger, discarding strings that resemble ordinary prose.

6. Scalable Implementation – The pipeline is built on Hadoop MapReduce for the counting phase and on Apache Spark’s MLlib for clustering. Map tasks independently process each file, emitting local n‑gram counts; Reduce tasks aggregate global frequencies. Spark’s in‑memory computation enables rapid clustering even on a multi‑node cluster (8 nodes), reducing total runtime to under an hour for the full dataset.

Experimental Results

• Coverage: The system correctly identified boiler‑plate in 92.3 % of the books.
• Precision: False‑positive rate was 2.7 %, mainly due to occasional mis‑classification of short, repetitive natural‑language sentences.
• Comparison with Rule‑Based Parsing: A handcrafted set of 30 regular expressions achieved only 78 % recall and 15 % false positives on the same corpus.
• Performance: On the Spark cluster, the entire workflow (frequency counting, candidate selection, clustering, and final removal) completed in ~45 minutes; a single‑machine implementation required ~4 hours.

Error Analysis – The approach struggled with non‑English introductions (French, German, etc.) and with author‑specific prefaces that do not repeat across many files. These cases were missed because they lack the high frequency signal required for detection. Additionally, heavily altered footers (e.g., “*** END OF THE PROJECT GUTENBERG EBOOK ***”) sometimes formed separate clusters, leading to multiple patterns for what is conceptually a single template.

Discussion – The statistical method dramatically reduces the maintenance burden associated with rule‑based parsers. It automatically adapts to new templates as long as they appear frequently enough in the corpus. However, the technique cannot fully replace human insight when the boiler‑plate carries semantic importance (e.g., author notes that should be retained for scholarly work). The authors suggest augmenting the pipeline with a supervised NLP classifier (e.g., BERT fine‑tuned to distinguish metadata from narrative) and incorporating a lightweight human‑in‑the‑loop verification step for low‑frequency or ambiguous sections.

Future Work – The paper outlines three main directions: (1) extending the approach to multilingual and multi‑format corpora (UTF‑8, HTML, PDF); (2) integrating semantic models to better differentiate meaningful introductions from boiler‑plate; and (3) developing an online, streaming version of the frequency estimator to handle continuously growing repositories without re‑processing the entire dataset.

In summary, the authors demonstrate that a simple yet powerful frequency‑based statistical model, combined with scalable big‑data processing frameworks, can reliably detect and strip boiler‑plate from massive collections of electronic texts. This reduces manual effort, improves downstream text‑analysis pipelines, and offers a flexible foundation for handling evolving template structures across diverse digital libraries.