Improved Relation Extraction with Feature-Rich Compositional Embedding Models

Compositional embedding models build a representation (or embedding) for a linguistic structure based on its component word embeddings. We propose a Feature-rich Compositional Embedding Model (FCM) for relation extraction that is expressive, generalizes to new domains, and is easy-to-implement. The key idea is to combine both (unlexicalized) hand-crafted features with learned word embeddings. The model is able to directly tackle the difficulties met by traditional compositional embeddings models, such as handling arbitrary types of sentence annotations and utilizing global information for composition. We test the proposed model on two relation extraction tasks, and demonstrate that our model outperforms both previous compositional models and traditional feature rich models on the ACE 2005 relation extraction task, and the SemEval 2010 relation classification task. The combination of our model and a log-linear classifier with hand-crafted features gives state-of-the-art results.

💡 Research Summary

The paper introduces the Feature‑rich Compositional Embedding Model (FCM), a novel approach to relation extraction that tightly integrates hand‑crafted, non‑lexical features with continuous word embeddings. Traditional relation extraction systems rely heavily on binary linguistic features (e.g., part‑of‑speech tags, dependency paths, entity types, positional cues) fed into log‑linear classifiers. While such features capture structural information, they suffer from poor generalization to unseen words. Conversely, recent embedding‑based methods represent words as dense vectors that generalize well but typically ignore the rich contextual role a word plays in a relation.

FCM bridges this gap by constructing a sub‑structure embedding for each token wᵢ as the outer product of its feature vector f(wᵢ) (a binary vector of size m) and its word embedding e(wᵢ) (a d‑dimensional real vector): h(wᵢ) = f(wᵢ) ⊗ e(wᵢ). This outer product yields an m × d matrix that simultaneously encodes “what the word is” (through e) and “how the word is used” (through f). All token‑level matrices are summed to form a sentence‑level embedding eₓ = Σᵢ h(wᵢ). The resulting representation is fed into a softmax layer that predicts the probability of each possible relation label y.

Training is performed end‑to‑end using stochastic gradient descent with AdaGrad. The loss is the standard cross‑entropy over the softmax distribution. Crucially, the word embeddings are fine‑tuned jointly with the feature‑to‑embedding transformation, allowing the model to adapt the semantic space to the specific demands of relation extraction. Because the outer product is mathematically equivalent to a second‑order polynomial feature combination, FCM can be viewed as a compact, learnable way to incorporate all pairwise interactions between features and embeddings without explicitly enumerating them.

The authors evaluate FCM on two widely used benchmarks. The ACE 2005 dataset presents a challenging cross‑domain setting: training data come from newswire, while test data span domains such as military, biomedical, and conversational text. Using only the standard set of 82 binary features (POS, dependency, entity type, etc.) and 50‑dimensional embeddings, FCM alone achieves 68.3 % F1, surpassing prior state‑of‑the‑art log‑linear and neural baselines. When combined with a traditional log‑linear classifier that also uses the same hand‑crafted features, the hybrid system reaches 71.2 % F1, establishing a new best result for coarse‑grained ACE relation extraction.

On SemEval‑2010 Task 8, which focuses on fine‑grained classification of a single relation per short sentence, FCM attains 85.6 % accuracy as a standalone model. Adding the log‑linear component pushes accuracy to 86.4 %, again beating the previously reported top scores. Detailed ablation studies show that the outer‑product construction contributes most of the gain: replacing it with a simple concatenation of features and embeddings reduces performance by several points, confirming that the multiplicative interaction is essential.

The paper also discusses practical aspects. Because each token’s sub‑structure embedding has size m × d, memory consumption can become significant when m (the number of binary features) is large. The authors mitigate this by limiting m to a few hundred and d to 50–200, but they acknowledge that large‑scale deployments would benefit from low‑rank approximations or feature selection. Moreover, all experiments assume gold‑standard entity boundaries; integrating an entity recognizer would introduce noise and likely lower scores, suggesting a direction for future end‑to‑end joint modeling.

In summary, the contributions are: (1) a mathematically clean method to fuse arbitrary linguistic features with word embeddings via outer product, (2) a log‑bilinear model that is both expressive and computationally efficient, (3) state‑of‑the‑art results on two major relation extraction benchmarks, and (4) an open‑source implementation that can be readily adapted to other structured prediction tasks. The work demonstrates that carefully combining traditional feature engineering with modern representation learning yields superior performance while preserving interpretability and flexibility.