An Efficient Interaction Human-AI Synergy System Bridging Visual Awareness and Large Language Model for Intensive Care Units

Intensive Care Units (ICUs) are critical environments characterized by high-stakes monitoring and complex data management. However, current practices often rely on manual data transcription and fragmented information systems, introducing potential risks to patient safety and operational efficiency. To address these issues, we propose a human-AI synergy system based on a cloud-edge-end architecture, which integrates visual-aware data extraction and semantic interaction mechanisms. Specifically, a visual-aware edge module non-invasively captures real-time physiological data from bedside monitors, reducing manual entry errors. To improve accessibility to fragmented data sources, a semantic interaction module, powered by a Large Language Model (LLM), enables physicians to perform efficient and intuitive voice-based queries over structured patient data. The hierarchical cloud-edge-end deployment ensures low-latency communication and scalable system performance. Our system reduces the cognitive burden on ICU nurses and physicians and demonstrates promising potential for broader applications in intelligent healthcare systems.

💡 Research Summary

The paper presents a comprehensive human‑AI synergy system designed for intensive care units (ICUs) that tackles two persistent challenges: manual transcription errors and fragmented information access. The architecture follows a cloud‑edge‑end paradigm. At the edge, a high‑resolution camera mounted on a servo‑controlled rig continuously captures bedside monitor screens without physical contact. A lightweight YOLOv5 detector identifies regions displaying vital signs (heart rate, blood pressure, SpO₂, etc.). Detected regions are fed into a CRNN‑based OCR pipeline with a CTC decoder, producing text strings that are subsequently normalized and mapped to standard clinical concepts. The extracted data are encoded in the Fast Healthcare Interoperability Resources (FHIR) format and transmitted to the cloud, dramatically reducing bandwidth usage and preserving patient privacy.

In the cloud, a large language model (LLM) powered interface enables physicians to query patient data using natural‑language voice commands. Automatic speech recognition (ASR) converts spoken input into text, which is transformed into prompts that guide the LLM to generate context‑aware SQL queries against the FHIR database. The LLM returns answers in natural language and can optionally render visualizations such as trend graphs. Prompt engineering ensures that the system understands clinical intent without requiring model retraining, keeping computational overhead low.

The three‑tier deployment (edge for acquisition, cloud for integration and LLM processing, end devices for interaction) provides low‑latency, scalable performance. Experimental evaluation shows screen‑region detection accuracy of 0.99, OCR accuracy above 96 %, and a 95 % reduction in transmitted data volume. LLM‑driven queries achieve an average response time of 1.2 seconds with 92 % answer correctness. Compared with traditional manual workflows, the system reduces transcription error rates by over 70 % and cuts physician data‑retrieval time by more than 60 %.

Overall, the study demonstrates that combining non‑invasive visual perception with semantic LLM interaction can significantly improve data reliability, reduce cognitive load, and enhance decision‑making efficiency in high‑stakes ICU environments. Future work may extend device compatibility, integrate predictive analytics, and explore broader clinical applications of the proposed human‑AI synergy framework.