Action-Sufficient Goal Representations

Hierarchical policies in offline goal-conditioned reinforcement learning (GCRL) addresses long-horizon tasks by decomposing control into high-level subgoal planning and low-level action execution. A critical design choice in such architectures is the goal representation-the compressed encoding of goals that serves as the interface between these levels. Existing approaches commonly derive goal representations while learning value functions, implicitly assuming that preserving information sufficient for value estimation is adequate for optimal control. We show that this assumption can fail, even when the value estimation is exact, as such representations may collapse goal states that need to be differentiated for action learning. To address this, we introduce an information-theoretic framework that defines action sufficiency, a condition on goal representations necessary for optimal action selection. We prove that value sufficiency does not imply action sufficiency and empirically verify that the latter is more strongly associated with control success in a discrete environment. We further demonstrate that standard log-loss training of low-level policies naturally induces action-sufficient representations. Our experimental results a popular benchmark demonstrate that our actor-derived representations consistently outperform representations learned via value estimation.

💡 Research Summary

This paper investigates a fundamental design issue in offline goal‑conditioned reinforcement learning (GCRL) when hierarchical policies are employed to solve long‑horizon tasks. In such architectures a high‑level planner proposes subgoals and a low‑level controller executes primitive actions to reach those subgoals. The interface between the two levels is a compressed goal representation. Existing hierarchical methods, most notably HIQL, learn this representation jointly with a value function, assuming that a representation sufficient for accurate value estimation is also sufficient for optimal control.

The authors challenge this assumption by introducing an information‑theoretic notion of action sufficiency. A representation Z = ϕ(S,G) is action‑sufficient if the conditional mutual information I(A; G | S,Z) equals zero, i.e., the optimal action distribution given the full goal G is identical to the distribution conditioned only on the compressed representation Z. They derive a decomposition of the conditional KL risk between the optimal policy and any policy that can only condition on Z: the risk splits into a modeling error term (which can be reduced by training) and a representation error term I(A; G | S,Z) that is irreducible unless the representation is action‑sufficient.

In parallel, they define value sufficiency as I(V; G | S,Z)=0, meaning the representation retains all information needed to reconstruct the optimal value function V(S,G). While value‑sufficient representations guarantee perfect value prediction, the paper proves that they do not guarantee action sufficiency. A simple one‑dimensional deterministic MDP illustrates the failure: an encoder that maps (S,G) to the absolute distance |S‑G| can recover the optimal value (which depends only on distance) but collapses opposite goals (left vs. right) into the same code, making it impossible for a policy to distinguish the correct action.

Empirically, the authors evaluate the two types of encoders on the cube task from OGBench. They use an oracle subgoal generator to eliminate planning errors and compare low‑level policies conditioned on (i) the value‑based encoder ϕ_V and (ii) an actor‑based encoder ϕ_A learned via advantage‑weighted regression (log‑loss). Despite the value function achieving a high order‑consistency ratio (indicating accurate learning), the low‑level policy using ϕ_V attains a dramatically lower success rate than the one using ϕ_A. This demonstrates that a representation can be perfectly aligned with the value function yet still lack critical information for action selection.

Further analysis shows that standard log‑loss training of the low‑level policy naturally yields an action‑sufficient representation, because the policy’s objective explicitly preserves the goal‑action relationship. Consequently, the authors argue that hierarchical GCRL systems should learn goal representations directly from the actor’s objective or incorporate explicit regularization that minimizes I(A; G | S,Z).

In summary, the paper makes three key contributions: (1) formalizes action sufficiency and shows its necessity for optimal low‑level control; (2) proves that value sufficiency does not imply action sufficiency, both theoretically and with concrete counterexamples; (3) provides empirical evidence that actor‑derived representations outperform value‑derived ones on a benchmark task. These findings suggest a shift in how goal representations are learned in offline hierarchical RL, emphasizing policy‑centric criteria over value‑centric ones to achieve reliable long‑horizon behavior.