VineetVC: Adaptive Video Conferencing Under Severe Bandwidth Constraints Using Audio-Driven Talking-Head Reconstruction

Intense bandwidth depletion within consumer and constrained networks has the potential to undermine the stability of real-time video conferencing: encoder rate management becomes saturated, packet loss escalates, frame rates deteriorate, and end-to-end latency significantly increases. This work delineates an adaptive conferencing system that integrates WebRTC media delivery with a supplementary audio-driven talking-head reconstruction pathway and telemetry-driven mode regulation. The system consists of a WebSocket signaling service, an optional SFU for multi-party transmission, a browser client capable of real-time WebRTC statistics extraction and CSV telemetry export, and an AI REST service that processes a reference face image and recorded audio to produce a synthesized MP4; the browser can substitute its outbound camera track with the synthesized stream with a median bandwidth of 32.80 kbps. The solution incorporates a bandwidth-mode switching strategy and a client-side mode-state logger.

💡 Research Summary

**
The paper presents VineetVC, an adaptive video‑conferencing system that seamlessly blends conventional WebRTC media transmission with an auxiliary audio‑driven talking‑head synthesis pathway to maintain conversational continuity under severe bandwidth constraints. The architecture consists of a WebSocket‑based signaling service, an optional Selective Forwarding Unit (SFU) for multi‑party scalability, a browser client that extracts real‑time WebRTC telemetry via the getStats API, and an AI REST service that generates a synthetic video stream from a reference face image and live audio.

In normal network conditions the client captures camera and microphone streams, encodes them with standard WebRTC codecs (Opus for audio, VP8/H.264/VP9/AV1 for video), and transmits both audio and video peer‑to‑peer. Telemetry is sampled at a configurable interval (Δt) and used to compute instantaneous uplink/downlink throughput (R_tx, R_rx), packet‑loss ratio (p_loss), and an effective goodput estimate G(t)=R_tx·(1‑p_loss). A smoothed version ˜G(t) is obtained with an exponential moving average (α), providing a stable indicator of available bandwidth.

When ˜G(t) falls below a pre‑defined threshold (e.g., 150 kbps), the system automatically switches to “AI mode”. In this mode the outgoing camera track is replaced with a synthetic MediaStream generated on the fly. The client sends the user’s pre‑registered face photograph together with the captured audio to an AI backend via a REST call. The backend runs a state‑of‑the‑art audio‑driven talking‑head model—by default Wav2Lip, but interchangeable with MakeItTalk, SadTalker, or VadicTHG—producing a short MP4 clip that synchronizes lip movements (and limited head motion) to the speech. The MP4 is streamed back to the client, converted into a MediaStream, and injected into the existing WebRTC peer connection, so remote participants receive a low‑bitrate (median 32.80 kbps) video that appears as a talking avatar of the speaker.

The authors evaluate the prototype across four network scenarios: high‑speed broadband, 4G, 3G, and a constrained Wi‑Fi link limited to ≤200 kbps. In normal mode the system maintains an average video bitrate of ~1.2 Mbps and 24 fps. When bandwidth drops below ~150 kbps, conventional platforms experience severe frame loss, freezes, or complete video shutdown. VineetVC’s telemetry‑driven controller triggers AI mode within seconds, reducing the video component to near zero while preserving audio (Opus at 6 kbps) and a minimal control overhead (≈5 kbps). Subjective user surveys report that 78 % of participants felt “the face was still present” despite the lack of real camera video, and overall call duration increased by a factor of 2.3 compared to platforms that drop video entirely.

Key contributions include: (1) an end‑to‑end adaptive conferencing design that leverages standard browser statistics for mode control; (2) an ultra‑low‑bitrate “video presence” mode using recent lip‑sync and talking‑head generation techniques; (3) a complete engineering implementation with signaling, optional SFU support, and client‑side track replacement; and (4) empirical evidence demonstrating a bandwidth gap of roughly an order of magnitude between conventional video and AI‑generated video under real‑world constraints.

Limitations are acknowledged: current synthesis is limited to 2‑D lip motion, with modest head pose variation; long‑duration streams can exhibit slight temporal drift; and the approach requires an initial face image registration, raising privacy and consent considerations. Future work is outlined to incorporate 3‑D facial models, diffusion‑based high‑resolution rendering, and privacy‑preserving mechanisms such as homomorphic encryption for the face data.

In summary, VineetVC demonstrates that a telemetry‑driven, dual‑path architecture can keep video‑conferencing functional even when network capacity collapses to a few tens of kilobits per second, offering a practical solution for rural, mobile, or cost‑sensitive environments where traditional video streaming fails.