Big Data and Cross-Document Coreference Resolution: Current State and Future Opportunities

Information Extraction (IE) is the task of automatically extracting structured information from unstructured/semi-structured machine-readable documents. Among various IE tasks, extracting actionable intelligence from ever-increasing amount of data depends critically upon Cross-Document Coreference Resolution (CDCR) - the task of identifying entity mentions across multiple documents that refer to the same underlying entity. Recently, document datasets of the order of peta-/tera-bytes has raised many challenges for performing effective CDCR such as scaling to large numbers of mentions and limited representational power. The problem of analysing such datasets is called “big data”. The aim of this paper is to provide readers with an understanding of the central concepts, subtasks, and the current state-of-the-art in CDCR process. We provide assessment of existing tools/techniques for CDCR subtasks and highlight big data challenges in each of them to help readers identify important and outstanding issues for further investigation. Finally, we provide concluding remarks and discuss possible directions for future work.

💡 Research Summary

The paper provides a comprehensive overview of Cross‑Document Coreference Resolution (CDCR) in the context of big‑data environments, outlining its fundamental concepts, sub‑tasks, state‑of‑the‑art techniques, and future research directions. It begins by emphasizing the explosion of unstructured textual sources—web pages, news articles, emails, and presentations—and the consequent need to resolve references that span multiple documents in order to build high‑quality knowledge bases, support semantic search, and enable question answering. CDCR is framed as a two‑stage pipeline: (1) Entity Identification, which corresponds to Named Entity Recognition (NER), and (2) Entity Classification, which determines whether pairs of mentions refer to the same real‑world entity.

The authors detail the traditional NER workflow, consisting of format analysis, tokenisation, gazetteer lookup, and grammar‑based rule application. They list widely used open‑source NER tools such as OpenNLP, Stanford NER, Illinois NER, and LingPipe, and discuss the challenges posed by metonymy, ambiguous names, and domain‑specific variations. For the classification stage, the paper surveys a broad spectrum of features—string similarity, lexical, syntactic, pattern‑based, count‑based, semantic, knowledge‑base, class‑based, list/inference/history, and relationship‑based—and explains how these are combined in supervised classifiers (SVM, maximum entropy, Hidden Markov Models) or unsupervised clustering algorithms (hierarchical clustering, DBSCAN).

A central focus of the work is the scalability problem that arises when the number of mentions grows to millions or billions, causing pairwise similarity computation to become O(N²). To address this, the authors discuss several mitigation strategies: candidate pruning, blocking techniques, locality‑sensitive hashing, and dimensionality reduction (PCA, random projection). They argue that these methods are essential for keeping both memory consumption and runtime within feasible limits.

The paper then shifts to big‑data challenges specific to CDCR. Large collections (terabytes to petabytes) introduce issues such as heterogeneous document formats (HTML, PDF, JSON), the need for distributed storage, and the difficulty of maintaining high precision while scaling. The authors propose a MapReduce‑based architecture built on Hadoop (and optionally Spark) that distributes the workload across three phases: (i) distributed ingestion and preprocessing on HDFS, (ii) local NER and feature extraction in mapper tasks, (iii) local clustering followed by a global merge in reducer tasks. An experimental prototype is described, demonstrating a twelve‑fold speedup on a 10 TB dataset compared with a single‑machine baseline, at the cost of a modest 2–3 % drop in F1 score.

In addition to the MapReduce solution, the paper surveys existing CDCR toolkits and libraries. It compares traditional NER systems, similarity search libraries such as FAISS and Annoy, and clustering frameworks like Mahout and MLlib. The authors also highlight recent advances that incorporate pre‑trained language models (BERT, RoBERTa) to generate contextual entity embeddings, which improve semantic similarity assessment and reduce reliance on handcrafted features. Knowledge‑base integration using DBpedia or Wikidata APIs is presented as a way to enrich the feature space with external facts.

The concluding section outlines a roadmap for future research. Key open problems include real‑time CDCR on streaming data, multimodal coreference that jointly processes text, images, and tables, leveraging deep contextual embeddings for richer similarity metrics, privacy‑preserving distributed learning (e.g., federated learning) for sensitive corpora, and the creation of large‑scale, high‑quality annotated benchmarks for systematic evaluation. The authors argue that progress in these areas will be pivotal for next‑generation applications such as dynamic knowledge graph construction, enterprise data integration, and large‑scale intelligence analysis.