When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints

Automated traffic continued to surpass human-generated traffic on the web, and a rising proportion of this automation was explicitly malicious. Evasive bots could pretend to be real users, even solve Captchas and mimic human interaction patterns. This work explores a less intrusive, protocol-level method: using TLS fingerprinting with the JA4 technique to tell apart bots from real users. Two gradient-boosted machine learning classifiers (XGBoost and CatBoost) were trained and evaluated on a dataset of real TLS fingerprints (JA4DB) after feature extraction, which derived informative signals from JA4 fingerprints that describe TLS handshake parameters. The CatBoost model performed better, achieving an AUC of 0.998 and an F1 score of 0.9734. It was accurate 0.9863 of the time on the test set. The XGBoost model showed almost similar results. Feature significance analyses identified JA4 components, especially ja4_b, cipher_count, and ext_count, as the most influential on model effectiveness. Future research will extend this method to new protocols, such as HTTP/3, and add additional device-fingerprinting features to test how well the system resists advanced bot evasion tactics.

💡 Research Summary

The paper investigates a protocol‑level approach to distinguishing malicious web bots from legitimate human users by leveraging TLS ClientHello fingerprints, specifically the JA4 format. Traditional bot‑detection techniques—CAPTCHAs, header spoofing, mouse‑movement analysis—are increasingly bypassed by AI‑driven bots that can mimic human behavior. In contrast, TLS handshakes expose low‑level implementation choices (cipher suites, extensions, ALPN order, TLS version) that are difficult to forge at scale and preserve user privacy because they do not require decryption of payload data.

The authors use the publicly available JA4DB repository, which aggregates 227,404 JA4 records collected via both controlled lab submissions and passive network monitoring. Records are labeled as “benign” (human traffic), “good bots” (well‑known crawlers such as Googlebot), and “bad bots” (malicious automation). Good bots are excluded from training; the final dataset contains 50,212 bad‑bot samples (≈22 % of total) and 148,610 benign samples (≈65 %). Labels are derived from the user‑agent string and application field, with any entry containing the term “bot” marked as malicious unless it matches a known crawler identifier.

Feature engineering parses each JA4 string into a set of 15 numeric and categorical variables, including ja4_b (TLS version indicator), ja4_s, ja4_h, cipher_count (number of advertised cipher suites), ext_count (number of extensions), and others. Missing values are imputed, categorical features are one‑hot encoded, and class‑imbalance is mitigated by applying sample weights during training.

Two gradient‑boosted decision‑tree models—XGBoost and CatBoost—are trained and tuned via five‑fold cross‑validation. Evaluation metrics include AUC, F1‑score, accuracy, precision, and recall. CatBoost achieves the best performance with an AUC of 0.998, F1 of 0.9734, and overall accuracy of 98.63 % on the held‑out test set; XGBoost follows closely with an AUC of 0.996. SHAP analysis reveals that ja4_b, cipher_count, and ext_count are the most influential features, indicating that malicious bots often use outdated or atypical TLS configurations compared with modern browsers.

The threat model outlines the protection scope: the method reliably detects non‑browser automation tools (e.g., Python requests, curl), header‑spoofing bots, and malicious agents employing custom TLS stacks. However, it cannot differentiate a human from a bot that drives a real browser engine (e.g., Puppeteer, Playwright) because such bots inherit the genuine browser’s JA4 fingerprint. Likewise, sophisticated TLS‑spoofing frameworks that deliberately craft ClientHello messages to match legitimate browser fingerprints can evade detection. Consequently, JA4 should be viewed as a strong supplemental signal rather than a standalone authentication mechanism.

Future work proposes extending the approach to newer transport protocols such as HTTP/3/QUIC, developing a JA4‑Q fingerprint for QUIC handshakes, and integrating additional device‑level fingerprints (OS, hardware, network stack) to build a multi‑modal detection system. The authors also plan to evaluate adversarial robustness by training bots to mimic browser JA4 patterns and to assess real‑time deployment considerations, including model compression and streaming inference.

In summary, the study demonstrates that TLS handshake‑level fingerprinting, when combined with modern gradient‑boosted classifiers, provides a highly accurate, privacy‑preserving method for detecting malicious web bots, while also highlighting its limitations against full‑stack browser automation and advanced TLS spoofing.