Not-in-Perspective: Towards Shielding Google's Perspective API Against Adversarial Negation Attacks

The rise of cyberbullying in social media platforms involving toxic comments has escalated the need for effective ways to monitor and moderate online interactions. Existing solutions of automated toxicity detection systems, are based on a machine or deep learning algorithms. However, statistics-based solutions are generally prone to adversarial attacks that contain logic based modifications such as negation in phrases and sentences. In that regard, we present a set of formal reasoning-based methodologies that wrap around existing machine learning toxicity detection systems. Acting as both pre-processing and post-processing steps, our formal reasoning wrapper helps alleviating the negation attack problems and significantly improves the accuracy and efficacy of toxicity scoring. We evaluate different variations of our wrapper on multiple machine learning models against a negation adversarial dataset. Experimental results highlight the improvement of hybrid (formal reasoning and machine-learning) methods against various purely statistical solutions.

💡 Research Summary

The paper addresses a critical weakness of Google’s Perspective API, namely its susceptibility to adversarial negation attacks where toxic sentences are prefixed with words such as “not” or “never”. Although Perspective relies on large‑scale machine‑learning models trained on massive corpora, these models treat words largely as statistical features and therefore fail to capture the logical inversion introduced by negation. Consequently, a sentence like “You are not an idiot” can still receive a high toxicity score, undermining automated moderation pipelines.

To mitigate this, the authors propose a hybrid defense architecture that wraps a formal reasoning module around any existing toxicity classifier, acting as both a pre‑processing and post‑processing layer. The workflow consists of three main stages: (1) Negation detection, where the input text is tokenized and parsed using Stanford’s Part‑of‑Speech tagger and a recursive syntactic parser. The parser isolates segments that are under the scope of a negation cue (e.g., “not”, “never”). (2) Two alternative correction strategies:
  a. Heuristic score adjustment – For each negated segment, the toxicity score returned by the underlying model (TS_i) is inverted as (1 − TS_i). The overall sentence score is recomputed as a weighted average of the original and inverted segment scores, with weights proportional to segment length (Equations 1‑4). This approach directly encodes the logical NOT operation while preserving the statistical backbone of the classifier.
  b. Antonym substitution and paraphrase expansion – Negated words are replaced by up to five context‑appropriate antonyms retrieved from Thesaurus/OneLook. Contextual sense disambiguation is performed with the Lesk algorithm to avoid semantic drift. The modified sentences are then fed to the Parrot paraphrasing model, which generates a diverse set of paraphrases that retain the original meaning. All paraphrases are scored by the toxicity classifier and the final toxicity estimate is the average of these scores. This method leverages semantic diversification to dilute the effect of a single negation cue.

The authors evaluate the framework on four toxicity models (BERT‑base, RoBERTa‑large, an LSTM, and a traditional SVM) using a curated adversarial dataset where each original toxic comment is transformed by inserting a negation cue. Results show that the heuristic adjustment reduces the average toxicity from 0.12 to 0.04 (≈66 % reduction) while slightly improving overall accuracy (78 % → 81 %). The antonym‑substitution/paraphrase pipeline achieves an even larger drop to 0.03 (≈75 % reduction) with only a modest 2 % increase in false‑positive rate. Processing latency remains low (≈150 ms per comment), indicating feasibility for real‑time deployment.

The paper also discusses limitations. The negation detector relies on correct syntactic parsing; informal language, slang, or emojis can degrade performance. Antonym selection may introduce meaning distortion, especially for polysemous words, despite Lesk‑based disambiguation. Parrot’s generation step is computationally intensive, requiring GPU resources; scaling to high‑traffic environments would need batching or lighter paraphrasing alternatives. Moreover, the study focuses exclusively on negation attacks and does not address more nuanced phenomena such as sarcasm or irony.

In conclusion, the work demonstrates that integrating formal logical reasoning with statistical machine‑learning models—embodying the “Learn2Reason” paradigm—significantly hardens toxicity detection systems against negation‑based adversarial attacks. The proposed wrapper is model‑agnostic, improves robustness without sacrificing speed, and offers a practical path for platforms that rely on Perspective or similar APIs. Future research directions include extending the framework to handle sarcasm, exploring lightweight post‑processing alternatives, and evaluating the approach on multilingual datasets.