Realistic Synthetic Household Data Generation at Scale

Advancements in foundation models have catalyzed research in Embodied AI to develop interactive agents capable of environmental reasoning and interaction. Developing such agents requires diverse, large-scale datasets. Prior frameworks generate synthetic data for long-term human-robot interactions but fail to model the bidirectional influence between human behavior and household environments. Our proposed generative framework creates household datasets at scale through loosely coupled generation of long-term human-robot interactions and environments. Human personas influence environment generation, while environment schematics and semantics shape human-robot interactions. The generated 3D data includes rich static context such as object and environment semantics, and temporal context capturing human and agent behaviors over extended periods. Our flexible tool allows users to define dataset characteristics via natural language prompts, enabling configuration of environment and human activity data through natural language specifications. The tool creates variations of user-defined configurations, enabling scalable data generation. We validate our framework through statistical evaluation using multi-modal embeddings and key metrics: cosine similarity, mutual information gain, intervention analysis, and iterative improvement validation. Statistical comparisons show good alignment with real-world datasets (HOMER) with cosine similarity (0.60), while synthetic datasets (Wang et al.) show moderate alignment (0.27). Intervention analysis across age, organization, and sleep pattern changes shows statistically significant effects (p < 0.001) with large effect sizes (Cohen’s d = 0.51-1.12), confirming bidirectional coupling translates persona traits into measurable environmental and behavioral differences. These contributions enable development and testing of household smart devices at scale.

💡 Research Summary

The paper addresses a critical bottleneck in embodied AI for household robotics: the lack of large‑scale, realistic datasets that capture the mutual influence between human behavior and home environments. Existing synthetic data pipelines treat environment generation and human activity synthesis as separate processes, which prevents models from learning the complex spatio‑temporal dependencies present in real homes. To overcome this, the authors propose a novel generative framework that tightly couples these two components through a bidirectional influence loop.

The system accepts structured natural‑language descriptions of household personas (age, occupation, hobbies, daily routines, etc.) and high‑level environmental constraints (house type, room layout). It then proceeds through four main modules:

1. Environment Schematic Generator – a large language model (LLM) creates a 3‑D layout with semantically annotated objects and rooms, guided by persona requirements. Visual floor‑plan sketches are produced to aid downstream steps.

2. Human Activity and HRI Generator – using a “least‑to‑most” prompt‑tuning strategy and a rolling‑window context mechanism, the LLM synthesizes temporally consistent activity sequences and human‑robot dialogues that respect the affordances of the generated environment.

3. Bidirectional Influence Controller – this orchestrator iteratively exchanges information between the environment and activity modules. For example, if a “laundry” activity appears, the controller requests the addition of a laundry basket and washing machine in the layout; conversely, the presence of a gaming console in the layout triggers the insertion of gaming sessions in the activity plan. The loop continues until convergence criteria are met (no further environment changes expected from activities and vice‑versa).

4. Universal Simulator Adapter – converts the intermediate representation into formats compatible with various simulators (e.g., Habitat, Isaac Gym), making the pipeline simulator‑agnostic while preserving semantic consistency.

Scalability is achieved by manipulating LLM sampling parameters (temperature, top‑p, top‑k) and by borrowing asset‑selection diversification techniques from the Holodeck system. This prevents mode collapse and ensures a rich variety of generated households.

For validation, the authors employ four quantitative metrics: (i) cosine similarity between multi‑modal embeddings of synthetic and real datasets, (ii) mutual information gain, (iii) intervention analysis (systematically varying persona attributes such as age, organization level, sleep pattern), and (iv) iterative improvement validation. Compared with the real‑world HOMER dataset, the synthetic data achieves a cosine similarity of 0.60, substantially higher than the 0.27 reported for prior synthetic approaches (Wang et al.). Intervention experiments show statistically significant effects (p < 0.001) with large effect sizes (Cohen’s d = 0.51–1.12), confirming that persona traits are faithfully reflected in both environmental configurations and behavioral patterns.

The paper’s contributions are: (1) a bidirectional, temporally consistent synthetic data generation framework for human‑robot interaction, (2) a persona‑driven environment synthesis method, (3) a novel coupling mechanism that grounds activities in environment semantics and vice‑versa, (4) comprehensive statistical validation demonstrating superior realism, and (5) preliminary Sim‑to‑Real experiments indicating practical applicability for training household robots.

Limitations include reliance on LLMs for geometry and texture fidelity, which may not guarantee physical plausibility, and a Sim‑to‑Real evaluation limited to a few simulated platforms. Future work will focus on tighter integration with physics engines, reinforcement‑learning‑based activity planners, and large‑scale real‑world deployments to further close the reality gap.