A Dictionary-based Approach to Racism Detection in Dutch Social Media

We present a dictionary-based approach to racism detection in Dutch social media comments, which were retrieved from two public Belgian social media sites likely to attract racist reactions. These comments were labeled as racist or non-racist by multiple annotators. For our approach, three discourse dictionaries were created: first, we created a dictionary by retrieving possibly racist and more neutral terms from the training data, and then augmenting these with more general words to remove some bias. A second dictionary was created through automatic expansion using a \texttt{word2vec} model trained on a large corpus of general Dutch text. Finally, a third dictionary was created by manually filtering out incorrect expansions. We trained multiple Support Vector Machines, using the distribution of words over the different categories in the dictionaries as features. The best-performing model used the manually cleaned dictionary and obtained an F-score of 0.46 for the racist class on a test set consisting of unseen Dutch comments, retrieved from the same sites used for the training set. The automated expansion of the dictionary only slightly boosted the model’s performance, and this increase in performance was not statistically significant. The fact that the coverage of the expanded dictionaries did increase indicates that the words that were automatically added did occur in the corpus, but were not able to meaningfully impact performance. The dictionaries, code, and the procedure for requesting the corpus are available at: https://github.com/clips/hades

💡 Research Summary

This paper investigates a dictionary‑based method for automatically detecting racist language in Dutch‑language social media comments. The authors collected comments from two publicly accessible Belgian Facebook pages that are known to attract racist reactions: one belonging to an anti‑Islamic organization and another to a right‑wing group. From the most recent 100 posts on each site they extracted 5,759 comments for training (4,880 from the first site, 879 from the second) and later gathered a separate test set of 1,116 comments (500 from the first site, 616 from the second). Each comment was annotated by three annotators with the labels “racist”, “non‑racist”, or “invalid” (the latter for non‑Dutch language or non‑textual content). Disagreements between the first two annotators were resolved by a third annotator. Inter‑annotator agreement measured by Cohen’s κ was 0.60 on the training set and 0.54 on the test set, indicating moderate but not high consistency, especially for the test data where the third annotator tended to label more comments as racist.

The core contribution is the construction of three discourse dictionaries that serve as the feature source for a Support Vector Machine (SVM) classifier. The first dictionary is built manually from the training data: terms that appear in racist comments are extracted, then grouped into semantic categories (e.g., skin colour, nationality, religion, migration, crime). For each category a neutral sub‑list (words that can be used in a non‑offensive way) and a negative sub‑list (explicit insults) are created. To reduce bias caused by the narrow domain of the source sites, the authors manually added extensive lists of countries, nationalities, and languages, and introduced wild‑cards to capture morphological variants and compound forms.

The second dictionary is an automatic expansion of the first one. A word2vec model trained on a large, general‑domain Dutch corpus is used to retrieve the ten most similar words for each entry in the manual dictionary. These candidate expansions are merged with the original lists, thereby increasing lexical coverage.

The third dictionary refines the automatically expanded version by human inspection: noisy or semantically unrelated terms introduced by word2vec are removed, yielding a “cleaned” dictionary.

For each comment, the authors compute the proportion of words that fall into each dictionary category, producing a feature vector of normalized frequencies (similar to the LIWC approach). These vectors are fed into a linear SVM; the regularization parameter C is tuned via five‑fold cross‑validation.

Experimental results show that the model using the manually cleaned dictionary achieves the best performance on the racist class, with an F‑score of 0.46, precision of 0.48 and recall of 0.44. The baseline model using only the original manual dictionary attains an F‑score of 0.42, while the model with the automatically expanded dictionary reaches 0.44. The overall accuracy across both classes is around 78 %, but the recall for the minority (racist) class remains modest, reflecting the difficulty of detecting relatively rare and context‑dependent hateful speech.

The authors note that the automatic expansion increased dictionary coverage by roughly 12 % but did not yield a statistically significant performance gain. This suggests that many of the added words, although present in the corpus, do not provide discriminative power for the racist vs. non‑racist distinction. They also discuss the limitations of their dataset: the two source pages are already biased toward anti‑Islamic and right‑wing discourse, so the corpus may not be representative of the broader Dutch‑speaking population. Moreover, the moderate κ scores highlight the inherent subjectivity in defining what constitutes a racist utterance, especially when cultural or regional nuances are involved (e.g., the term “zwartzak” having a historical meaning unrelated to skin colour).

In the discussion, the paper suggests several avenues for future work. One is to incorporate more diverse data sources to reduce domain bias. Another is to explore richer contextual embeddings (e.g., BERT for Dutch) for dictionary expansion, which could capture subtler semantic relations than word2vec. Adding syntactic features such as part‑of‑speech tags, dependency relations, or sentiment scores could also improve detection, as prior work has shown that pronoun usage, modality, and negative emotion words are salient markers of racist discourse. Finally, improving annotation protocols—perhaps by providing more detailed guidelines, training sessions, or involving annotators from varied cultural backgrounds—could raise inter‑annotator agreement and produce a more reliable gold standard.

In summary, the study demonstrates that a carefully crafted, domain‑specific dictionary combined with simple frequency‑based features can achieve reasonable performance in detecting racist language in Dutch social media. While the approach is transparent and computationally lightweight, its effectiveness is limited by the quality of the dictionary, the representativeness of the data, and the inherent ambiguity of the target phenomenon. The paper contributes valuable resources (the dictionaries, code, and a reproducible data‑request procedure) to the community and lays groundwork for more sophisticated, context‑aware hate‑speech detection systems in low‑resource languages.