V-SHiNE: A Virtual Smart Home Framework for Explainability Evaluation

Explanations are essential for helping users interpret and trust autonomous smart-home decisions, yet evaluating their quality and impact remains methodologically difficult in this domain. V-SHiNE addresses this gap: a browser-based smarthome simulation framework for scalable and realistic assessment of explanations. It allows researchers to configure environments, simulate behaviors, and plug in custom explanation engines, with flexible delivery modes and rich interaction logging. A study with 159 participants demonstrates its feasibility. V-SHiNE provides a lightweight, reproducible platform for advancing user-centered evaluation of explainable intelligent systems

💡 Research Summary

The paper presents V‑SHiNE (Virtual Smart Home with Intelligent and Explainable Features), a browser‑based simulation platform designed to enable scalable, realistic, and reproducible user‑centered evaluation of explanations for autonomous smart‑home systems. The authors begin by outlining the challenge: modern smart homes integrate numerous cyber‑physical devices and machine‑learning‑driven automation, which creates opaque decision‑making that users must trust. Traditional evaluation methods—physical testbeds, small‑scale lab studies, or purely questionnaire‑based online surveys—are either costly, low‑fidelity, or lack ecological validity. V‑SHiNE addresses this gap by providing a fully configurable virtual environment where researchers can define rooms, devices, automation rules, and contextual variables using JSON specifications.

The system architecture consists of a React‑Phaser front‑end for first‑person navigation, a Next.js back‑end with Socket.IO for real‑time event handling, and a MongoDB store for persistent logging. The front‑end renders interactive objects based on the W3C Thing Description (TD) model, allowing automatic generation of control panels for any device type. Automation rules follow an “if‑then” schema and are evaluated on the server whenever relevant state changes occur. Triggers (time‑based, failure events) and task definitions guide participants through experimental flows.

A key contribution is the flexible explanation subsystem. Explanations can be supplied directly from static templates within V‑SHiNE or fetched dynamically from external explanation engines via REST or WebSocket APIs. Researchers can experiment with three delivery modalities—push (automatic), pull (on‑demand), and interactive (chat‑style query)—simply by adjusting a configuration file, without touching core code. This design enables systematic comparison of different explanation strategies under identical environmental conditions.

Data collection is comprehensive: every interaction (device toggles, navigation clicks, rule evaluations), every explanation shown, and every user rating (thumb up/down) are logged with timestamps and session identifiers. The resulting dataset links subjective perception of explanation quality with objective performance metrics such as task completion time and error rates, supporting rigorous statistical and machine‑learning analyses.

To ensure reliability, the authors built an extensive Vite‑based test suite comprising unit tests for API endpoints, integration tests for real‑time Socket.IO workflows, and schema validation tests for user‑provided JSON files. With roughly 500 test cases across 21 files, the suite achieves over 70 % coverage of branches, functions, lines, and statements, providing confidence that the platform behaves consistently across deployments.

The platform’s feasibility is demonstrated through a large‑scale user study involving 159 participants. Two explanation conditions were compared: a static, context‑independent explanation and a context‑aware explanation that adapts to the current state of the virtual home (e.g., temperature, window status). Participants performed a series of everyday smart‑home tasks (turning devices on/off, investigating unexpected behavior) while receiving explanations in the assigned modality. Results indicated that context‑aware explanations significantly improved perceived usability, trust, and task efficiency, confirming that V‑SHiNE can capture meaningful differences in explanation quality.

Additionally, the authors re‑implemented a scenario from the CIRCE study—a previously system‑level causal explanation method—within V‑SHiNE. This illustrates how existing explanation techniques that lack user‑centered validation can be seamlessly migrated into a realistic, task‑based experimental setting.

In conclusion, V‑SHiNE offers a lightweight, Docker‑deployable, MIT‑licensed framework that lowers the barrier for conducting reproducible, ecologically valid user studies of explainable AI in smart homes and broader CPS contexts. Future work will extend the platform to support multi‑user interactions, richer device models, and longitudinal deployments, further broadening its applicability for the HCI and AI research communities.