Voice-Driven Semantic Perception for UAV-Assisted Emergency Networks

Unmanned Aerial Vehicle (UAV)-assisted networks are increasingly foreseen as a promising approach for emergency response, providing rapid, flexible, and resilient communications in environments where terrestrial infrastructure is degraded or unavailable. In such scenarios, voice radio communications remain essential for first responders due to their robustness; however, their unstructured nature prevents direct integration with automated UAV-assisted network management. This paper proposes SIREN, an AI-driven framework that enables voice-driven perception for UAV-assisted networks. By integrating Automatic Speech Recognition (ASR) with Large Language Model (LLM)-based semantic extraction and Natural Language Processing (NLP) validation, SIREN converts emergency voice traffic into structured, machine-readable information, including responding units, location references, emergency severity, and Quality-of-Service (QoS) requirements. SIREN is evaluated using synthetic emergency scenarios with controlled variations in language, speaker count, background noise, and message complexity. The results demonstrate robust transcription and reliable semantic extraction across diverse operating conditions, while highlighting speaker diarization and geographic ambiguity as the main limiting factors. These findings establish the feasibility of voice-driven situational awareness for UAV-assisted networks and show a practical foundation for human-in-the-loop decision support and adaptive network management in emergency response operations.

💡 Research Summary

The paper introduces SIREN, an AI‑driven framework that converts unstructured voice radio communications from first‑responders into structured, machine‑readable semantic data for use in UAV‑assisted emergency networking. The authors argue that while UAVs can rapidly deploy resilient communication links in disaster zones, the voice channel—still the most reliable means for on‑scene coordination—remains a “black box” for automated network management because it lacks a formal representation. Recent breakthroughs in Automatic Speech Recognition (ASR) and Large Language Models (LLMs) enable the extraction of actionable information from speech, but no prior work has integrated these technologies specifically for UAV positioning, bandwidth allocation, or other adaptive network decisions in emergency scenarios.

SIREN’s architecture consists of three sequential stages. First, an ASR module transcribes raw audio. The system can switch between a local Whisper model for offline, connectivity‑constrained operation and a cloud‑based Assembly API that offers higher accuracy, speaker diarization, and sentiment scores. Second, the transcribed text is fed to an LLM (LLaMA 3.2 deployed via Ollama) that is prompted with a strict JSON schema defining the required fields: locations, responding units, emergency severity level, and QoS requirements (e.g., video or image bandwidth). The LLM’s probabilistic output is then passed through a deterministic validation layer comprising three NLP components: (1) Named Entity Recognition (SpaCy) to verify geographic entities, (2) speaker diarization results to align detected speakers with extracted unit identifiers, and (3) sentiment analysis to cross‑check the LLM’s severity classification, escalating or de‑escalating the level when sentiment indicates panic or calmness. Finally, validated location names are geocoded with Geopy and presented in a web‑based interactive map (Folium) alongside the structured JSON payload. The output is deliberately limited to information explicitly mentioned in the audio, avoiding any speculative inference.

Because publicly available emergency audio corpora are scarce, the authors built a synthetic dataset. They used an LLM to generate realistic multi‑speaker emergency dialogues, then synthesized speech with ElevenLabs TTS, employing distinct voice profiles for each responder. Five scenarios were created, varying in language (English vs. Portuguese), number of speakers (four vs. six), dialogue length, and background noise level. Scenarios 1–3 represent low to medium complexity, while scenarios 4–5 introduce linguistic challenges (Portuguese pronunciation, ambiguous place names) and higher speaker counts. Both clean and noisy versions were produced, enabling robustness testing.

Experiments were run on a single laptop (Ryzen 5 5600H, RTX 3050, Ubuntu 22.04). Transcription quality was measured with Word Error Rate (WER). Whisper achieved 7.2 % WER on clean audio, while the Assembly API reduced it to 4.5 %. Under noisy conditions, WER rose to 12 % (Whisper) and 8 % (Assembly), confirming the advantage of cloud‑based models in adverse acoustic environments. Semantic extraction performance was evaluated with recall/precision metrics: location recall ≈ 92 %, unit F1 ≈ 0.88, and emergency‑level accuracy ≈ 0.91 across all scenarios. Even with added noise, the system maintained > 80 % accuracy for the core entities. The main failure modes were speaker diarization mismatches (≈ 15 % of cases) leading to incorrect unit‑speaker attribution, and geographic ambiguity in Portuguese scenarios where multiple places share the same name, causing occasional geocoding errors.

The authors highlight several strengths of SIREN. Its modular design allows operators to select lightweight local models when bandwidth is limited or to switch to high‑accuracy cloud services when connectivity permits. The JSON schema ensures that downstream UAV control algorithms (e.g., dynamic positioning, adaptive bandwidth allocation) can ingest the data without custom parsers. By providing a human‑in‑the‑loop visualization, the framework supports both automated decision‑making and operator oversight, bridging the gap between raw voice chatter and actionable network policies.

Limitations are acknowledged. Speaker diarization accuracy remains a bottleneck, especially with overlapping speech or low‑quality microphones. Geographic ambiguity could be mitigated by integrating a pre‑compiled gazetteer or leveraging additional sensor modalities (e.g., GPS‑tagged radios). The current evaluation relies on synthetic audio; real‑world field trials are necessary to validate performance under true emergency conditions, including variable channel fading, background sirens, and unpredictable speech patterns.

Future work will focus on (1) real‑time streaming implementation with sub‑second latency, (2) multimodal fusion of audio with video or sensor data to enrich context, (3) deployment in live disaster drills to assess operational impact, (4) advanced speaker localization using microphone arrays, and (5) tighter integration with GIS databases to resolve ambiguous place names. The authors envision SIREN as a foundational perception layer that empowers UAV‑assisted networks to adapt on the fly, prioritize critical traffic, and ultimately improve the effectiveness of emergency response operations.