Beyond Bias Scores: Unmasking Vacuous Neutrality in Small Language Models

The rapid adoption of Small Language Models (SLMs) for resource constrained applications has outpaced our understanding of their ethical and fairness implications. To address this gap, we introduce the Vacuous Neutrality Framework (VaNeu), a multi-dimensional evaluation paradigm designed to assess SLM fairness prior to deployment. The framework examines model robustness across four stages - biases, utility, ambiguity handling, and positional bias over diverse social bias categories. To the best of our knowledge, this work presents the first large-scale audit of SLMs in the 0.5-5B parameter range, an overlooked “middle tier” between BERT-class encoders and flagship LLMs. We evaluate nine widely used SLMs spanning four model families under both ambiguous and disambiguated contexts. Our findings show that models demonstrating low bias in early stages often fail subsequent evaluations, revealing hidden vulnerabilities and unreliable reasoning. These results underscore the need for a more comprehensive understanding of fairness and reliability in SLMs, and position the proposed framework as a principled tool for responsible deployment in socially sensitive settings.

💡 Research Summary

The paper addresses a critical gap in the ethical and fairness evaluation of Small Language Models (SLMs) that occupy the 0.5‑5 B parameter range—a “middle tier” that is increasingly deployed in resource‑constrained settings such as edge devices, mobile applications, and low‑power servers. While large language models (LLMs) have been extensively audited for bias, and very small models (e.g., BERT‑base) have been studied for fairness, the intermediate‑scale SLMs have received little systematic scrutiny despite their growing practical importance.

To fill this gap, the authors propose the Vacuous Neutrality Framework (VaNeu), a four‑stage, multi‑dimensional evaluation paradigm that simultaneously measures: (1) Bias, using established bias scores from benchmarks such as BBQ, StereoSet, and CrowS‑Pairs; (2) Utility, i.e., task competence, measured by F1 on BBQ and Language Modeling Score (LMS) on the other two benchmarks; (3) Ambiguity Handling, which quantifies a model’s ability to abstain (“Unknown”) when the input is underspecified, using Target‑to‑NonTarget Ratio (TNR) and Unknown Ratio (UR); and (4) Positional Bias, which captures systematic preferences for particular answer positions (A, B, C, …) via a normalized KL‑divergence metric.

The central concept introduced is “vacuous neutrality”: a failure mode where a model appears unbiased because it receives a low bias score, yet this neutrality is achieved through degenerate behavior such as random guessing, over‑abstention, or reliance on superficial heuristics (e.g., always picking the first option). In such cases, low bias co‑exists with poor utility, mis‑calibrated uncertainty, or strong positional shortcuts, rendering the model unreliable for real‑world deployment.

Experimental Setup
The authors evaluate nine open‑source, instruction‑tuned SLMs from four families: Qwen2.5, LLaMA3.2, Gemma3, and Phi. Models are grouped into “Tiny” (0.5‑2 B) and “Small” (2‑4 B) tiers. All evaluations are performed zero‑shot in a multiple‑choice format with a fixed greedy decoder (temperature = 0, top‑p = 1.0) to eliminate sampling variance. Each experiment is repeated ten times with different random shuffles of demographic instances to ensure robustness.

Three socially sensitive benchmarks are unified into a multiple‑choice QA format:

• BBQ (Bias Benchmark for QA) – provides both bias scores and ground‑truth answers, and includes ambiguous cases where “Unknown” is the correct response.
• StereoSet – focuses on stereotypical vs. anti‑stereotypical completions; bias is measured via Stereo Score, while utility is captured by LMS.
• CrowS‑Pairs – a minimal‑pair dataset that isolates stereotype polarity; bias is again measured by Stereo Score.

Key Findings

1. Bias vs. Utility Mismatch – Several Tiny models (e.g., Qwen2.5‑0.5B) achieve near‑zero bias scores but suffer dramatically low F1/LMS, indicating that they are essentially guessing or over‑abstaining. Conversely, some Small models (e.g., LLaMA3.2‑3B) maintain modest bias while delivering high utility, disproving the assumption that fairness necessarily trades off with performance.

2. Ambiguity Handling Divergence – Models differ markedly in UR and TNR. High‑performing models correctly abstain on ambiguous inputs (high UR) while still making accurate target predictions on clear cases (high TNR). Others either never abstain (low UR, high over‑commitment) or abstain everywhere (UR ≈ 1, TNR ≈ 0), both undesirable.

3. Positional Bias Patterns – Phi‑Mini variants exhibit strong positional bias despite low overall bias scores, consistently favoring option A. This reveals reliance on surface patterns rather than semantic reasoning and suggests that positional bias is an orthogonal failure mode that can hide behind favorable bias metrics.

4. Impact of Compression Techniques – Models that have undergone pruning or quantization sometimes show reduced bias scores, but this often coincides with increased UR or amplified positional bias, highlighting that compression is not fairness‑neutral.

5. Scale Effects – Across families, increasing parameter count generally improves utility and reduces extreme vacuous neutrality, yet the relationship is not monotonic; certain 2 B models outperform some 3 B counterparts on ambiguity handling, indicating that architecture and training data matter as much as size.

Implications
The study demonstrates that a single bias metric is insufficient for trustworthy deployment of SLMs. VaNeu’s four‑stage assessment uncovers hidden vulnerabilities that would be missed by conventional audits. Practitioners are urged to adopt a holistic evaluation pipeline: first screen for overt bias, then verify that the model can solve the target task, handle uncertainty responsibly, and avoid superficial answer‑position shortcuts.

Contributions

1. Introduction of the Vacuous Neutrality Framework, a principled, task‑agnostic, dataset‑agnostic suite of metrics.
2. The first large‑scale, systematic audit of nine SLMs in the 0.5‑5 B range across three major bias benchmarks.
3. Empirical evidence that bias, utility, ambiguity handling, and positional bias are largely independent dimensions, each requiring separate scrutiny.
4. Open‑source release of code, evaluation scripts, and processed benchmark splits to facilitate reproducibility and future research.

In summary, the paper provides a comprehensive methodology and empirical baseline for assessing fairness and reliability in mid‑scale language models, warning against the deceptive comfort of low bias scores and advocating for multi‑dimensional audits before real‑world deployment.