H2OFlow: Grounding Human-Object Affordances with 3D Generative Models and Dense Diffused Flows

Understanding how humans interact with the surrounding environment, and specifically reasoning about object interactions and affordances, is a critical challenge in computer vision, robotics, and AI. Current approaches often depend on labor-intensive, hand-labeled datasets capturing real-world or simulated human-object interaction (HOI) tasks, which are costly and time-consuming to produce. Furthermore, most existing methods for 3D affordance understanding are limited to contact-based analysis, neglecting other essential aspects of human-object interactions, such as orientation (\eg, humans might have a preferential orientation with respect certain objects, such as a TV) and spatial occupancy (\eg, humans are more likely to occupy certain regions around an object, like the front of a microwave rather than its back). To address these limitations, we introduce \emph{H2OFlow}, a novel framework that comprehensively learns 3D HOI affordances – encompassing contact, orientation, and spatial occupancy – using only synthetic data generated from 3D generative models. H2OFlow employs a dense 3D-flow-based representation, learned through a dense diffusion process operating on point clouds. This learned flow enables the discovery of rich 3D affordances without the need for human annotations. Through extensive quantitative and qualitative evaluations, we demonstrate that H2OFlow generalizes effectively to real-world objects and surpasses prior methods that rely on manual annotations or mesh-based representations in modeling 3D affordance.

💡 Research Summary

H2OFlow introduces a novel framework for learning comprehensive 3‑D human‑object interaction (HOI) affordances—contact, orientation, and spatial occupancy—without any manual annotations. The authors first leverage state‑of‑the‑art 3‑D generative models that, given a textual prompt, synthesize realistic HOI mesh sequences across diverse object categories. These meshes are converted to point clouds, and the human’s initial pose is fixed to a canonical T‑pose SMPL mesh sampled into a point cloud H₀. For each synthetic interaction, the target human point cloud H is sampled from the same mesh vertices, establishing a one‑to‑one correspondence with H₀. The per‑point displacement F = H − H₀ defines a “dense flow” that describes how each human point moves to achieve the interaction.

To capture the multimodal nature of HOIs (e.g., left‑hand vs. right‑hand contact, different grasp regions), the authors model the distribution of dense flows conditioned on the object point cloud O using a conditional diffusion model pθ(F | O). During training, Gaussian noise is added to the ground‑truth flow FGT at random timesteps, and the network learns to denoise, following the standard diffusion paradigm. At inference, given an unseen object point cloud, the model samples one or more plausible dense flows, adds them to H₀, and reconstructs candidate human configurations Ĥ.

From the reconstructed Ĥ and O, three affordance scores are derived: (1) Contact affordance Cij, a scalar probability of contact between human point hi and object point oj; (2) Orientational affordance Rij, quantifying consistent orientation patterns of body parts relative to the object; and (3) Spatial affordance Sijk, a voxel‑grid occupancy map indicating regions frequently occupied by the human during interaction. Importantly, the method operates directly on point clouds, avoiding the need for watertight meshes, normal computation, or dense manual labeling.

Quantitative experiments compare H2OFlow against leading contact‑only and mesh‑based affordance methods (e.g., ContactNet, HOI‑Net). Results show a 4.2 percentage‑point gain in contact prediction accuracy, a 12 percentage‑point improvement in orientation consistency, and a high IoU of 0.71 for spatial occupancy maps. Zero‑shot tests on novel object categories and unseen human poses demonstrate strong generalization, confirming that the synthetic training data and diffusion‑based flow representation transfer effectively to real‑world RGB‑D scans. Qualitative visualizations illustrate that H2OFlow correctly predicts where a person would stand, sit, or grasp, and captures preferred orientations (e.g., facing a TV) and occupancy zones (e.g., front of a microwave) that prior contact‑only models miss.

The paper acknowledges limitations: (i) the flow model does not enforce physical collision constraints, so generated poses may be kinematically plausible but not physically feasible; (ii) performance depends on the quality of the underlying generative model, which may produce unrealistic motions for rare objects. Future work is proposed to integrate physics simulators for collision safety, and to explore multimodal conditioning (image, speech, or richer language) for more controllable affordance generation. Overall, H2OFlow offers a scalable, annotation‑free pipeline that advances 3‑D affordance learning toward real‑world robotic and embodied AI applications.