MSM-BD: Multimodal Social Media Bot Detection Using Heterogeneous Information

Although social bots can be engineered for constructive applications, their potential for misuse in manipulative schemes and malware distribution cannot be overlooked. This dichotomy underscores the critical need to detect social bots on social media platforms. Advances in artificial intelligence have improved the abilities of social bots, allowing them to generate content that is almost indistinguishable from human-created content. These advancements require the development of more advanced detection techniques to accurately identify these automated entities. Given the heterogeneous information landscape on social media, spanning images, texts, and user statistical features, we propose MSM-BD, a Multimodal Social Media Bot Detection approach using heterogeneous information. MSM-BD incorporates specialized encoders for heterogeneous information and introduces a cross-modal fusion technology, Cross-Modal Residual Cross-Attention (CMRCA), to enhance detection accuracy. We validate the effectiveness of our model through extensive experiments using the TwiBot-22 dataset.

💡 Research Summary

**
The paper introduces MSM‑BD, a multimodal social media bot detection framework that simultaneously leverages profile images, user statistical features, and tweet text to improve detection accuracy over traditional single‑modality approaches. The architecture consists of three specialized encoders: a visual encoder based on a pretrained ResNet‑18 to extract fine‑grained image representations; a user‑feature textual encoder that applies feature engineering, linear transformation, and GELU activation to a rich set of metadata (followers, followees, tweet counts, account age, etc.); and a tweets textual encoder that feeds each of the N most recent tweets into MiniLM, a lightweight pretrained language model, followed by a Sentence SE Fusion (SSEF) module. SSEF combines a multilayer perceptron (MLP) for dimensionality reduction with a Transformer encoder to capture inter‑tweet dependencies, yielding a compact yet expressive tweet embedding.

The core contribution is the Cross‑Modal Residual Cross‑Attention (CMRCA) module. After aligning the three modality embeddings to a common dimension, CMRCA assigns each modality a specific role as Value (V), Query (Q), or Key (K) in three separate cross‑attention streams. For example, user features act as V, tweet embeddings as Q, and image embeddings as K. Each stream uses multi‑head attention, and the resulting heads are concatenated and linearly projected. To preserve original modality information and mitigate over‑fitting, residual connections concatenate the raw embeddings with their attention‑enhanced counterparts. A final multi‑head attention layer integrates the three residual‑augmented streams, and a linear classifier outputs the bot probability.

Experiments are conducted on the publicly available TwiBot‑22 dataset, which contains one million Twitter accounts (≈86 % humans, 14 % bots) with rich multimodal annotations. Using the standard train/validation/test splits, MSM‑BD achieves an accuracy of 0.8002 and an F1‑score of 0.6105, surpassing all listed baselines such as BotRGCN (0.7966/0.5750), SGBot (0.7508/0.3659), and other feature‑based or graph‑based methods. The authors attribute the performance gain to the explicit cross‑modal interaction facilitated by CMRCA and the residual design that retains modality‑specific cues.

Strengths of the work include: (1) a clear motivation for multimodal fusion in bot detection; (2) a well‑structured encoder pipeline that balances expressive power (ResNet‑18, MiniLM) with computational efficiency; (3) the novel CMRCA mechanism that systematically defines V/Q/K roles, enabling richer information exchange than naïve concatenation; and (4) thorough evaluation on a large‑scale, realistic dataset with fair comparison to prior art.

However, several limitations are noted. The user‑feature encoder relies on handcrafted feature engineering, which may not generalize to other platforms without substantial redesign. The inclusion of a full ResNet‑18 and multi‑head attention layers increases GPU memory consumption, potentially hindering real‑time deployment. Moreover, the current model does not incorporate graph‑structured network information (the “G” modality), which has proven useful in prior bot detection research; integrating such data could further boost performance. Finally, the paper provides limited analysis of model interpretability—understanding which modality contributes most to a particular decision would be valuable for practitioners.

Future directions suggested include: (a) replacing ResNet‑18 with a more lightweight visual backbone (e.g., MobileNet‑V3) to reduce inference cost; (b) extending CMRCA to fuse graph embeddings alongside image and text; (c) exploring self‑supervised multimodal pretraining to lessen reliance on labeled bot data; and (d) applying domain adaptation techniques to transfer the framework to other social media ecosystems such as Instagram or TikTok.

In conclusion, MSM‑BD demonstrates that carefully designed cross‑modal attention mechanisms can substantially improve bot detection performance on heterogeneous social media data. While the approach sets a new state‑of‑the‑art on TwiBot‑22, addressing computational efficiency and modality generalization will be key to broader adoption in real‑world moderation systems.