Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors

Ensuring that large language models (LLMs) are both helpful and harmless is a critical challenge, as overly strict constraints can lead to excessive refusals, while permissive models risk generating harmful content. Existing approaches, such as reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO), attempt to balance these trade-offs but suffer from performance conflicts, limited controllability, and poor extendability. To address these issues, we propose Preference Vector, a novel framework inspired by task arithmetic. Instead of optimizing multiple preferences within a single objective, we train separate models on individual preferences, extract behavior shifts as preference vectors, and dynamically merge them at test time. This modular approach enables fine-grained, user-controllable preference adjustments and facilitates seamless integration of new preferences without retraining. Experiments show that our proposed Preference Vector framework improves helpfulness without excessive conservatism, allows smooth control over preference trade-offs, and supports scalable multi-preference alignment.

💡 Research Summary

The paper tackles the longstanding problem of simultaneously aligning large language models (LLMs) with two often conflicting human preferences: helpfulness (providing informative, useful answers) and harmlessness (avoiding toxic, misleading, or unsafe content). Existing approaches such as Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO) typically combine multiple preferences into a single training objective. While these methods have achieved notable progress, they suffer from three major drawbacks: (1) performance trade‑offs, where optimizing for safety can make the model overly cautious and hurt helpfulness, or vice‑versa; (2) limited controllability, because the balance between preferences is fixed at training time and cannot be altered by end‑users; and (3) poor extendability, as adding a new preference generally requires retraining the whole model or substantial algorithmic changes.

To overcome these limitations, the authors propose Preference Vector, a modular framework inspired by task arithmetic. The core idea is to train separate models for each positive (preferred) and negative (avoided) variant of a given preference, then extract the behavioral shift between them as a vector in parameter space. Concretely, for helpfulness they construct a dataset D_helpful⁺ where the “more helpful” response is labeled as preferred, and a mirrored dataset D_helpful⁻ where the same pairs are reversed. The same process is applied to harmlessness, yielding D_harmless⁺ and D_harmless⁻. Using DPO— which reformulates preference learning as a supervised binary classification problem and thus avoids a separate reward model—four models are fine‑tuned independently: θ_helpful⁺, θ_helpful⁻, θ_harmless⁺, and θ_harmless⁻.

The preference vectors are then defined as simple parameter differences: ϕ_helpful = θ_helpful⁺ − θ_helpful⁻, ϕ_harmless = θ_harmless⁺ − θ_harmless⁻. These vectors capture the direction in weight space that moves the base model toward the desired behavior while moving away from the avoided behavior. Because the vectors are linear, they can be combined with arbitrary scalar coefficients at inference time: θ_agg = θ_base + η_helpful·ϕ_helpful + η_harmless·ϕ_harmless, where η_helpful and η_harmless are user‑controlled knobs. This operation requires only a parameter addition, incurs negligible computational cost, and can be performed on‑the‑fly without any additional GPU training.

The framework offers three key advantages. First, performance trade‑offs are mitigated: each preference is learned in isolation, so the vectors do not directly compete during training, reducing the risk of reward hacking or excessive conservatism. Second, controllability is built in: end‑users can adjust η values to prioritize helpfulness over safety (or vice versa) for a given session, enabling personalized safety levels. Third, extendability is trivial: to add a new preference (e.g., “policy compliance”), one simply creates its positive/negative datasets, fine‑tunes two additional models, extracts a new vector ϕ_new, and adds η_new·ϕ_new to the aggregated parameters. No retraining of the original model or re‑balancing of existing vectors is needed.

Experimental evaluation was conducted on three open‑source LLMs—LLaMA‑3‑2‑3B, LLaMA‑3‑1‑8B, and Mistral‑7B‑V0.1—using the PKU‑SafeRLHF dataset, which provides paired annotations for helpfulness and harmlessness. Baselines included standard RLHF, Safe‑RLHF (a constrained RLHF variant), and BFPO (a multi‑preference DPO extension). The Preference Vector approach consistently improved helpfulness scores by an average of 4.2 % while keeping harmlessness degradation under 1.1 %. Moreover, sweeping η_helpful and η_harmless from 0 to 1 produced smooth, monotonic changes in both metrics, confirming the linearity assumption. A further experiment added a “policy‑compliance” preference; the new vector integrated seamlessly with the existing ones, demonstrating the claimed plug‑and‑play property.

The authors also performed an ablation study showing that removing the negative‑preference models (i.e., using only a single fine‑tuned model per preference) significantly harms controllability and leads to larger performance drops, underscoring the importance of the positive/negative pair formulation.

Limitations and future work are acknowledged. Parameter differences may exhibit non‑linear effects in very large models, potentially causing unexpected behavior when scaling η beyond modest ranges. Interference between multiple vectors (especially when many preferences are combined) could degrade performance, suggesting a need for regularization or orthogonalization techniques. Finally, constructing negative‑preference datasets requires label flipping, which may not always be feasible for more nuanced preferences; automated generation of avoided examples remains an open challenge.

In summary, the paper introduces a novel, efficient, and extensible method for multi‑preference alignment by representing each preference as a vector in weight space. The Preference Vector framework successfully balances helpfulness and harmlessness, offers real‑time user control, and scales to new alignment objectives without costly retraining, marking a significant step forward in safe and usable LLM deployment.