We investigate whether agents can instead learn an internal alignment embedding (IAE): a differentiable latent variable that tracks predicted externalized harm and shapes policy gradients toward harm reduction through three key properties:

1. Predictive: IAE forecasts alignment-relevant outcomes via counterfactual reasoning over candidate actions.

2. Regulatory: IAE modulates perception and action selection through attention gating and gradient coupling.

3. Distributed: IAE propagates across agent neighborhoods via graph diffusion with controllable similarity weighting.

improved performance through learned relational representations. Wang et al. (2021) propose QPLEX with duplex dueling and attention mechanisms for value factorization, demonstrating improved stability. Liu et al. (2023) develop NA2Q using neural attention additive models for interpretable multi-agent Q-learning, showing that attention can provide both performance and interpretability.

However, existing work does not integrate graph mechanisms with internal alignment states that track predicted harm.

ESAI integrates graph diffusion with IAE dynamics, enabling similarity-weighted propagation of alignment-relevant information. The diffusion operator propagates IAE across neighborhoods, while similarity weighting modulates edge strengths based on learned agent identities. The biasmitigation regularizer provides interpretable fairness-performance tradeoffs not present in standard graph communication architectures.

Differentiable memory systems such as Neural Turing Machines (Graves et al., 2014) enable learned read/write operations for temporal credit assignment, showing that memory mechanisms can be trained end-to-end via gradient descent. Miconi et al. (2018) demonstrate that backpropagation can train differentiable Hebbian plasticity rules that adapt network weights during deployment, showing benefits for continual learning and adaptation. This work proves that associative learning rules can be optimized jointly with network parameters.

However, no existing work couples Hebbian learning with alignment-specific internal states or uses Hebbian traces to support counterfactual forecasting of harm. Standard memory architectures (attention over episodic buffers, recurrent states) do not encode associative traces between internal alignment embeddings and perceptual features.

ESAI couples Hebbian traces to IAE dynamics via differentiable read operations, enabling alignmentaware memory updates that support counterfactual forecasting. The Hebbian matrix encodes coactivation history of IAE and percepts, providing context for harm prediction while preserving end-to-end trainability.

Recent work explores using vision-language models to provide semantic guidance for reinforcement learning. Wu et al. (2025) propose using VLMs as action advisors for online RL, demonstrating that pre-trained models can provide interpretable action suggestions. However, this approach requires external model queries at each timestep rather than learning intrinsic alignment representations embedded in the agent’s parameters.

ESAI differs by learning alignment representations end-to-end without requiring external model queries, pre-trained components, or oracle access to semantic models during deployment.

Potential-based reward shaping (Ng et al., 1999) adds Φ(s t+1 ) -Φ(s t ) to rewards, preserving optimal policies under certain conditions and providing theoretical guarantees.

ESAI differs in three fundamental ways:

1. Learned dynamics: IAE evolves via learned function g ϕ and graph diffusion operator L, not hand-designed potential functions. This enables adaptation to environment-specific harm structures without manual redesign.

2. Counterfactual supervision: IAE is trained to predict harm outcomes via counterfactual forecasting, not approximate task value. This separates alignment objectives from task rewards.

3. Perceptual gating: IAE modulates perception via attention, enabling non-Markovian salience modulation unavailable to potential-based methods that only augment rewards.

ESAI trades the theoretical guarantees of potential-based shaping (policy invariance) for adaptive capacity and richer internal dynamics, though this tradeoff requires empirical investigation.

While individual components exist in isolation-counterfactual reasoning for credit assignment (Foerster et al., 2018;Zhou et al., 2022), graph diffusion for communication (Jiang et al., 2020), attention for coordination (Iqbal and Sha, 2019;Liu et al., 2020;Wang et al., 2021;Liu et al., 2023), external shields (Alshiekh et al., 2018;Chatterji et al., 2025), intrinsic social preferences (Hughes et al., 2018;Alamiyan-Harandi et al., 2023), ethical environment design (Mayoral-Macau et al., 2025), and differentiable memory (Graves et al., 2014;Miconi et al., 2018) 3 Embedded Safety-Aligned Intelligence: Framework Definition Definition 1 (Internal Alignment Embedding). An internal alignment embedding (IAE) is a differentiable latent variable E t ∈ R k maintained by each agent that satisfies:

1. Predictive correspondence: E t correlates with predicted externalized harm under learned dynamics.

2. Gradient coupling: E t influences policy gradients via differentiable transformations of rewards or observations.

3. Temporal persistence: E t evolves through learned update rules that preserve information across multiple timesteps.

Definition 2 (Embedded Safety-Aligned Intelligence). A multi-agent learning system exhibits Embedded Safety-Aligned Intelligence if:

1. Each agent maintains an IAE satisfying Definition 1.

2. Policy learning incorporates a differentiable alignment objective that penalizes predicted harm encoded in IAE.

3. IAE dynamics are supervised (implicitly or explicitly) to forecast alignment-relevant outcomes.

4. The system supports distributed coordination through IAE propagation across agent neighborhoods.

ESAI systems differ from external alignment mechanisms (reward shaping, constraints, shields) in that alignment pressure arises from internal learned representations rather than external supervisory signals. This embedding has three potential advantages:

• Gradient-based adaptation: Differentiability enables online learning of alignment objectives without discrete switching or constraint satisfaction solvers.

• Perceptual salience: IAE can gate attention to alignment-relevant features, potentially improving sample efficiency in sparse-harm environments.

• Distributed coordination: Graph diffusion of IAE enables decentralized alignment pressure without centralized oversight.

However, ESAI also introduces challenges: (1) IAE semantics depend on quality of harm supervision, (2) learned embeddings may lack interpretability, and (3) computational overhead increases with embedding dimension and graph connectivity. We formalize these tradeoffs in subsequent sections.

We present one possible instantiation of the ESAI framework. Alternative architectures satisfying Definitions 1-2 may exist; this section illustrates design principles rather than prescribing a unique implementation.

Each agent augments a standard policy-gradient learner with an internal alignment embedding E t ∈ R k . All computations involving E t are differentiable, enabling gradient propagation into policy parameters θ.

(1)

Assumption 1 (Harm Observability). There exists a measurable harm signal h t : S ×A N ×S → R ≥0 satisfying:

1. Exogeneity: h t is defined independently of policy parameters θ 2. Observability: h t is computable from transition (s t , a t , s t+1 )

This assumption encodes domain-specific harm definitions as exogenous inputs. ESAI does not learn what constitutes harm-only how to predict and avoid externally specified harm. The normative content of h t must be provided by system designers and is subject to the biases discussed in Sec. 8.

Each agent i ∈ {1, . . . , N } maintains IAE E i,t ∈ R k . We define an alignment potential:

where [s t ; E i,t ] denotes concatenation and σ is a nonlinear activation. The alignment objective encourages low-norm embeddings:

This objective encodes the principle that lower IAE magnitude corresponds to lower predicted harm-a design choice that must be validated empirically.

The embedding for agent i evolves via:

where:

• γ E ∈ [0, 1) controls temporal persistence

The diffusion term -α j∈N (i) L ij E j,t propagates alignment-relevant information across agent neighborhoods. This design enables decentralized coordination: agents in similar states with high graph connectivity will have correlated IAE dynamics.

For each candidate action a ∈ A, agent i forecasts the next-step IAE:

where h ψ is a learned forecast network and read(H i,t ) provides Hebbian memory context (defined in Sec. 5.6).

Forecast supervision. The forecast network h ψ is trained to predict actual next-step IAE via the loss:

where E i,t+1 is computed from Eq. (4) using the executed action a i,t .

We scalarize forecasted harm via:

To avoid non-differentiable arg min, we define a softmin reference distribution with temperature τ :

The expected reference embedding is:

The differentiable alignment regret penalizes deviations from the harm-minimizing reference:

where the second term uses lagged neighbor embeddings E j,t (not future E j,t+1 ) to maintain causal consistency.

Temperature annealing. We propose annealing τ from high to low values over training for three theoretical reasons:

1. Early exploration: High τ smooths π ref , preventing premature convergence when forecasts are inaccurate. 2. Gradient stability: Soft targets reduce gradient variance compared to hard arg min. 3. Late discretization: As τ → 0, π ref recovers near-deterministic behavior, sharpening alignment signals.

Predictor stability via EMA target network. To prevent predictor-policy collusion, we propose maintaining an exponential moving average (EMA) target network:

with τ ema ≈ 0.995. All counterfactual forecasts E (a)

i,t+1 are computed using h ψtarget to stabilize gradient flow, analogous to stability mechanisms in self-supervised learning and double Q-learning.

We transform extrinsic rewards with the alignment penalty:

Advantages A i,t are computed using generalized advantage estimation (GAE) on r ′ i,t . The policy loss follows PPO-Clip:

where

Implementation note. The PPO-Clip formulation above is one possible instantiation; ESAI principles are compatible with alternative policy gradient methods (A2C, TRPO, SAC, MPO). Appendix C provides a complete training loop using PPO as an illustrative-not prescriptive-example.

Attention weights are computed from the current IAE to bias perceptual salience. To ensure dimensional consistency, we use projection matrix W a ∈ R d×k :

where ⊙ denotes element-wise product and z i,t ∈ R d is the observation vector.

Theoretical motivation: In sparse-harm environments, agents must attend to low-probability events (e.g., victim states). IAE-weighted attention provides a differentiable mechanism for learned salience modulation that could improve sample efficiency compared to uniform attention-though this remains an empirical question.

A Hebbian memory matrix H i,t ∈ R k×d updates via outer-product learning:

where δ H > 0 ensures decay and η H is the learning rate.

The Hebbian trace supports counterfactual forecasting via differentiable read:

where vec(•) flattens the matrix. This read vector enters forecast model h ψ in Eq. ( 5).

Theoretical motivation: Hebbian traces encode historical co-activations of IAE and perceptual features, providing context for counterfactual prediction. This could improve temporal credit assignment by associating past states with delayed harm outcomes.

The graph Laplacian L in Eq. ( 4) is row-normalized with spectral radius ρ(L) ≤ 2. Diffusion weights are modulated by cosine similarity of learned identity embeddings ϕ i ∈ R did :

To mitigate emergent in-group favoritism, we introduce a similarity-suppression regularizer:

where S is the similarity matrix with entries S ij = β ij and à is the weighted adjacency.

Theoretical motivation: Similarity-weighted diffusion enables faster propagation among similar agents, potentially accelerating coordination. However, this also creates risk of in-group bias. The regularizer L bias provides a tunable fairness-performance tradeoff.

The full objective combines policy loss, entropy regularization, and alignment penalties:

where β is entropy coefficient, λ H regularizes Hebbian trace norms, λ D controls Laplacian smoothness, and λ bias governs fairness.

Proposition 1 (Bounded IAE Under Contraction Condition). Consider the IAE update in Eq. (4). Assume:

3. There exists B 0 such that ∥g ϕ (0, 0, 0)∥ 2 ≤ B 0 4. The spectral condition holds:

Then sup t ∥E i,t ∥ 2 < ∞ for all agents i and trajectories.

Proof sketch. By Lipschitz continuity and boundedness:

Taking norms in Eq. ( 4):

Then iterating:

The spectral condition can be enforced via spectral normalization of L and gradient clipping on g ϕ .

Remark 1 (Boundedness Does Not Imply Alignment). Proposition 1 guarantees that IAE remains bounded, preventing numerical instability. However, bounded E i,t is a necessary but not sufficient condition for aligned behavior. Convergence to socially optimal equilibria-where agents jointly minimize harm while maximizing task performance-remains an open theoretical problem. The stability result ensures the learning process is well-defined; it does not guarantee the learned policy is aligned.

Proposition 2 (Hebbian Trace Stability). If the state-IAE joint distribution admits bounded second moments

holds, then the Hebbian trace converges in mean-square sense to a bounded fixed point

Proof sketch. Taking expectations and norms in Eq. ( 15):

] by Cauchy-Schwarz, this forms a contraction with fixed point:

Full proof with mean-square convergence is in Appendix E.

The ESAI forward pass involves per agent: The framework integrates four mechanisms: (1) differentiable counterfactual alignment penalties enabling gradient-based harm forecasting, (2) IAE-weighted attention biasing perceptual salience, (3) Hebbian affect-memory coupling supporting temporal credit assignment, and (4) similarityweighted graph diffusion with bias-mitigation controls. We derived conditions for bounded internal embeddings under Lipschitz constraints and spectral radius bounds, analyzed computational complexity as O(N |A|kd), and discussed theoretical properties including contraction dynamics and fairness-performance tradeoffs.

• Formal definition of internal alignment embeddings and embedded safety-aligned intelligence (Definitions 1-2)

• Differentiable counterfactual alignment penalty via softmin reference distributions

Consider the gradient of alignment regret (Eq. 10) with respect to forecast parameters ψ:

(21)

From Eq. ( 9):

The softmin gradient is:

where

i,t+1 ∥ 2 . For the norm gradient:

Combining these terms:

This decomposition shows that gradients flow through both the forecast magnitudes R(a) and the predicted embeddings E (a)

i,t+1 , enabling joint learning of harm prediction and embedding dynamics. The temperature τ controls the sharpness of the gradient signal: low τ produces sparse gradients concentrated on the minimum-harm action, while high τ distributes gradients across multiple actions.

To satisfy the spectral condition in Proposition 1, we enforce:

The spectral radius is:

Case 1: γ E ≥ αλ N . In this case, µ i ≥ 0 for all i, so:

since λ 2 ≈ 0 for connected graphs. The spectral condition becomes γ E < 1 -L g .

Case 2: γ E < αλ N . The largest magnitude eigenvalue is:

requiring:

Practical implementation. We normalize L via spectral normalization to ensure λ N ≤ 2, then choose:

For typical values γ E = 0.9, L g = 0.05, ρ max = 0.95, this gives α < min{0.925, 0.025} = 0.025.

To enforce L g -Lipschitz continuity of the IAE update function g ϕ , we use spectral normalization of all weight matrices in the MLP. For a two-layer network:

the Lipschitz constant is bounded by:

where L σ is the Lipschitz constant of the activation function σ.

For ReLU activations, L σ = 1, so we enforce:

This is achieved via spectral normalization: replace W i with W i / max(∥W i ∥ 2 , L g ) after each gradient update.

The ESAI framework (Definitions 1-2) admits alternative implementations beyond the architecture presented in Sec. 4. We describe three variants:

Replace deterministic E i,t with stochastic E i,t ∼ q ϕ (E | z i,t , a i,t ) and optimize a variational lower bound:

where p prior is a standard Gaussian prior.

Advantages:

• Uncertainty quantification over harm predictions

• Robust to noisy observations via posterior averaging

Disadvantages:

• Increased computational cost (reparameterization trick, KL computation)

• Additional hyperparameter β (KL weight)

• May increase variance in policy gradients

Replace MLP forecast network h ψ with a transformer operating over action sequences:

enabling multi-step lookahead counterfactuals over horizon H.

Formulation: The transformer takes as input a sequence of length H + 1:

• Token 0: Current observation z i,t (embedded via learned projection)

• Tokens 1 to H: Candidate action sequence a 1 , . . . , a H

Output is the predicted IAE after executing the action sequence:

), where [0] denotes the first output token.

• Captures long-horizon harm accumulation • Attention mechanism provides interpretability over action sequences • Can model complex temporal dependencies Disadvantages:

• Exponential complexity O(|A| H ) for horizon H > 1 • Requires significant training data for stable learning • Increased memory footprint for attention matrices

For continuous action spaces a ∈ R m , replace discrete softmin (Eq. 8) with continuous distribution:

where

i,t+1 ∥ 2 as before. The expected reference embedding becomes:

Approximation via sampling: Since the integral is intractable, approximate via Monte Carlo sampling:

where a k ∼ q(a) are samples from a proposal distribution q (e.g., current policy).

Advantages:

• Extends ESAI to continuous control domains

• Maintains differentiability via reparameterization trick

• Scales to high-dimensional action spaces Disadvantages:

• Requires careful choice of proposal distribution q

• Variance in Monte Carlo estimate can destabilize training

• May need adaptive temperature schedules for convergence C Appendix C: Illustrative Implementation (Non-Normative)

Disclaimer. This appendix provides one possible instantiation of the ESAI framework using PPO-Clip. The specific algorithmic choices (PPO vs. other policy gradient methods, hyperparameter values, update schedules) are illustrative, not prescriptive. Alternative implementations satisfying Definitions 1-2 are equally valid. Empirical validation is required before adopting any specific configuration.

Typical hyperparameter values for the algorithm are shown in Table 2. Initialize: Forecast network h ψ , target h ψtarget ← h ψ 3: Initialize: IAE E i,0 ← 0, Hebbian H i,0 ← 0 for all i 4: Initialize: Identity embeddings {ϕ i } N i=1 , graph Laplacian L 5: // All expectations and norms assumed finite; stability conditions in Sec. 6 6: for episode m = 1 to M do Compute IAE-weighted attention:

// Counterfactual forecasting (conceptual; may use sampling for large |A|) 16:

for each a ∈ A do 17:

Forecast:

Expected reference embedding:

Execute joint action: a t = (a 1,t , . . . , a N,t )

Observe transition: s t+1 , rewards {r ext i,t } N i=1 24:

for agent i = 1 to N do 25:

// Update IAE via learned dynamics g ϕ and graph diffusion 26:

// Update Hebbian associative trace 28:

// Compute alignment regret (first term: deviation from reference; second: neighbor regularization) Update policy:

// Forecast network update (supervised on realized transitions)

// IAE dynamics update via backpropagation 42:

Update IAE dynamics:

// Auxiliary updates 44:

Update EMA target:

Anneal temperature: τ m+1 ← max(τ min , τ 0 exp(-m/K τ ))

46:

// Graph update: recompute normalized Laplacian from similarity-weighted adjacency 47:

Update similarity weights: β ij ← max(0, cos(ϕ i , ϕ j )) for all i, j 48:

Recompute graph Laplacian:

Apply bias regularization: Update {ϕ i } via ∇ ϕi L bias 50: end for Ensure: Trained policy π θ , IAE dynamics g ϕ , forecast network h ψ Interpretive notes. The IAE encodes a learned internal proxy for alignment-relevant externalities, trained via counterfactual consistency and policy gradients rather than explicit ground-truth harm labels. The forecast network h ψ is trained on realized transitions and queried counterfactually at inference time to construct the reference distribution π ref . The enumeration over actions in line 14 is conceptual; in practice, this may be approximated via top-K sampling or restricted candidate sets for large action spaces. D Appendix D: Hyperparameter Sensitivity Analysis (Theoretical) Disclaimer. This appendix presents theoretical conjectures about hyperparameter sensitivity based on mathematical properties of the ESAI dynamics. These are not empirically validated recommendations. The analysis below identifies expected qualitative behaviors; quantitative guidance requires experimental validation across diverse environments.

We discuss expected sensitivity to key hyperparameters based on theoretical considerations:

Information-theoretic perspective. The minimal IAE dimension required to represent alignmentrelevant structure is bounded by the mutual information between observations and harm outcomes:

where I(Z; H) is the mutual information between observation distribution Z and harm variable H.

Expected behavior:

•

Trade-off analysis. The diffusion term -αLE i,t balances two competing objectives:

• Coordination (favors high α): Faster propagation of alignment signals across agent neighborhoods • Individuality (favors low α): Preserves agent-specific alignment states adapted to local contexts Spectral constraint. Proposition 1 requires:

For γ E = 0.9, L g = 0.05, λ max (L) = 2, this gives upper bound α < 0.025.

For δ H = 0.02, this gives τ mem = 50 timesteps.

Stability constraint. Proposition 2 requires:

For bounded states ∥z i,t ∥ 2 ≤ 10 and IAE ∥E i,t ∥ 2 ≤ 5 (typical after convergence):

With η H = 10 -3 , we need δ H > 0.05, but we use δ H = 0.02 to allow longer memory. This requires monitoring ∥H i,t ∥ F for potential instability.

Expected behavior: E Appendix E: Proof of Hebbian Stability (Complete)

We provide the complete proof of Hebbian trace convergence in mean-square sense. Theorem 1 (Hebbian Trace Convergence). Consider the Hebbian update in Eq. ( 15):

Assume:

1. The joint process (E i,t , z i,t ) is ergodic with stationary distribution µ 2. Second moments are bounded:

where:

Proof. Define the error matrix ∆ i,t = H i,t -H * i . Subtracting the fixed point from the update equation:

Substituting the fixed point expression

where

] is zero-mean under the stationary distribution.

First moment convergence. Taking expectations under the stationary distribution:

Second moment convergence. Taking squared Frobenius norms:

where ⟨•, •⟩ F is the Frobenius inner product.

Taking expectations and using ergodicity to decouple ∆ i,t (depends on history) from ξ i,t (current):

For the noise variance term:

Thus the second-moment recursion becomes:

This is a contraction with fixed point:

For the steady-state error to remain bounded and small, we require:

which gives the design constraint:

For ϵ tol = η H C E C z , this simplifies to:

Exponential convergence rate. From the recursion:

The convergence rate is ρ = (1 -δ H ) 2 , giving exponential decay with time constant:

For δ H = 0.02, convergence time is τ conv ≈ 50 timesteps.

Practical implications. This theorem guarantees that Hebbian traces converge to a stable statistical summary H * i encoding the expected co-activation pattern of IAE and observations under the policy’s stationary distribution. The design constraint δ H > η H C E C z can be verified via:

We provide theoretical estimates of computational overhead for ESAI mechanisms relative to standard PPO baselines. 2. Hierarchical alignment. Extend to hierarchical policies with alignment embeddings at multiple temporal scales:

• High-level IAE E high t for strategic harm (long-horizon)

• Low-level IAE E low t for tactical harm (short-horizon)

• Hierarchical diffusion between levels

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.Versus constrained optimization. CPO(Achiam et al., 2017) enforces hard safety constraints via trust regions. ESAI trades hard guarantees for soft, differentiable penalties that enable gradientbased learning. In high-dimensional action spaces where constraint satisfaction is expensive, ESAI’s continuous relaxation may offer computational advantages-but at the cost of occasional constraint violations.Versus multi-objective RL. Multi-objective methods optimize Pareto frontiers over task and safety objectives. ESAI implicitly performs multi-objective optimization via λ reg (Eq. 12) but adds internal dynamics (attention, diffusion) unavailable to standard scalarization approaches. Whether this added complexity provides practical benefits is an empirical question.8 Limitations and AssumptionsDependence on harm specification.Computational overhead. Counterfactual forecasting requires |A| forward passes per decision, increasing wall-clock time. For large action spaces, approximate methods (top-K sampling, learned action proposals) are needed but may sacrifice alignment quality.

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.Versus constrained optimization. CPO(Achiam et al., 2017) enforces hard safety constraints via trust regions. ESAI trades hard guarantees for soft, differentiable penalties that enable gradientbased learning. In high-dimensional action spaces where constraint satisfaction is expensive, ESAI’s continuous relaxation may offer computational advantages-but at the cost of occasional constraint violations.Versus multi-objective RL. Multi-objective methods optimize Pareto frontiers over task and safety objectives. ESAI implicitly performs multi-objective optimization via λ reg (Eq. 12) but adds internal dynamics (attention, diffusion) unavailable to standard scalarization approaches. Whether this added complexity provides practical benefits is an empirical question.8 Limitations and AssumptionsDependence on harm specification.

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.Versus constrained optimization. CPO(Achiam et al., 2017) enforces hard safety constraints via trust regions. ESAI trades hard guarantees for soft, differentiable penalties that enable gradientbased learning. In high-dimensional action spaces where constraint satisfaction is expensive, ESAI’s continuous relaxation may offer computational advantages-but at the cost of occasional constraint violations.Versus multi-objective RL. Multi-objective methods optimize Pareto frontiers over task and safety objectives. ESAI implicitly performs multi-objective optimization via λ reg (Eq. 12) but adds internal dynamics (attention, diffusion) unavailable to standard scalarization approaches. Whether this added complexity provides practical benefits is an empirical question.8 Limitations and Assumptions

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.Versus constrained optimization. CPO(Achiam et al., 2017) enforces hard safety constraints via trust regions. ESAI trades hard guarantees for soft, differentiable penalties that enable gradientbased learning. In high-dimensional action spaces where constraint satisfaction is expensive, ESAI’s continuous relaxation may offer computational advantages-but at the cost of occasional constraint violations.Versus multi-objective RL. Multi-objective methods optimize Pareto frontiers over task and safety objectives. ESAI implicitly performs multi-objective optimization via λ reg (Eq. 12) but adds internal dynamics (attention, diffusion) unavailable to standard scalarization approaches. Whether this added complexity provides practical benefits is an empirical question.

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.Versus constrained optimization. CPO(Achiam et al., 2017) enforces hard safety constraints via trust regions. ESAI trades hard guarantees for soft, differentiable penalties that enable gradientbased learning. In high-dimensional action spaces where constraint satisfaction is expensive, ESAI’s continuous relaxation may offer computational advantages-but at the cost of occasional constraint violations.

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.Versus constrained optimization. CPO(Achiam et al., 2017)

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.Versus constrained optimization. CPO

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999) provides theoretical guarantees (policy invariance under certain conditions) that ESAI lacks. However, ESAI’s learned dynamics could adapt to non-stationary harm structures where hand-designed potentials fail. The tradeoff between theoretical guarantees and adaptive capacity requires empirical investigation.

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping(Ng et al., 1999)

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative ParadigmsVersus reward shaping. Potential-based shaping

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?7.3 Comparison to Alternative Paradigms

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.Interpretability limits. While IAE magnitude is designed to correlate with harm, individual latent dimensions may lack semantic meaning. Can we derive conditions under which IAE factors into interpretable subspaces (e.g., via disentanglement objectives)?

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.Decentralized coordination. Graph diffusion of IAE enables distributed alignment pressure: agents in connected neighborhoods will develop correlated harm predictions without centralized oversight. This could improve scalability compared to centralized critics, but introduces new failure modes under adversarial graph topology.

Attention gating: O(kd) 4. Hebbian update: O(kd) 5. Graph diffusion: O(|N (i)|k) For bounded-degree graphs with |N (i)| = O(1) and N agents, total complexity is O(N |A|k 2 + N |A|kd) per step. Counterfactual forecasting dominates; this can be mitigated via top-K action sampling, reducing |A| to K ≪ |A|.

• Integration of IAE with attention gating, Hebbian memory, and graph diffusion • Stability analysis and complexity bounds (Propositions 1-2)• Identification of open theoretical questions and empirical validation requirements Limitations.• Interpretability: Develop methods to ground IAE semantics (e.g., via correlation with domain harm metrics, interventional probes, disentanglement objectives)• Robustness: Characterize performance under distribution shift, adversarial perturbations, and graph topology attacks • Scalability: ϕ , graph diffusion L, and Hebbian memory H t . Counterfactual forecasts (computed via EMA target network ψ target ) generate alignment penalties AR t that shape policy gradients. IAEweighted attention α t modulates perceptual input z t . Dotted lines indicate gradient flow; solid lines denote forward computation.

• Integration of IAE with attention gating, Hebbian memory, and graph diffusion • Stability analysis and complexity bounds (Propositions 1-2)• Identification of open theoretical questions and empirical validation requirements Limitations.• Interpretability: Develop methods to ground IAE semantics (e.g., via correlation with domain harm metrics, interventional probes, disentanglement objectives)

• Integration of IAE with attention gating, Hebbian memory, and graph diffusion • Stability analysis and complexity bounds (Propositions 1-2)• Identification of open theoretical questions and empirical validation requirements Limitations.

• Integration of IAE with attention gating, Hebbian memory, and graph diffusion • Stability analysis and complexity bounds (Propositions 1-2)

Algorithm 1 presents the complete ESAI training loop for multi-agent systems. Each episode consists of environment interaction, internal state updates, and network optimization. The forecast network is trained on realized transitions and queried counterfactually at inference time. Counterfactual supervision arises from minimizing the discrepancy between realized IAE trajectories and a softminweighted reference over forecasted counterfactual IAEs.

• Too low (λ reg → 0): Policy ignores alignment, maximizes task reward, high harm • Optimal (λ reg ≈ 0.1): Balanced task performance and alignment • Too high (λ reg > 1): Over-conservative policies, poor task performance, extreme risk-aversion