An Architecture-Led Hybrid Report on Body Language Detection Project

A defensible report must therefore be architecture-led: it should explain what the model is doing “under the hood” and then translate that into system-level interface decisions and limitations.

The BodyLanguageDetection project [1] uses VLMs to convert sampled video frames into machine-consumable artifacts: per-frame person detections, pixel-space bounding boxes, and 1 arXiv:2512.23028v1 [cs.CV] 28 Dec 2025 prompt-conditioned per-person attributes (emotion by default), plus an optional annotated video.

Two models play different roles: Qwen2.5-VL-7B-Instruct is used for batch, schema-oriented extraction; Llama-4-Scout-17B-16E-Instruct is used for interactive, single-frame inspection and prompt iteration.

The contributions of this report are:

• An architecture-led explanation of how multimodal Transformers represent images as tokens and fuse them with text instructions using attention [2,3];

• A model-specific discussion of Qwen2.5-VL-7B-Instruct and Llama-4-Scout-17B-16E-Instruct grounded in public documentation [4,5,6];

This section summarizes only the system context needed to motivate architectural choices. The BodyLanguageDetection pipeline [1] transforms an uploaded video into structured artifacts through four stages:

1. Frame sampling: Videos are not processed at native frame rate. Instead, frames are sampled at a configurable interval using OpenCV [7]. Sampling reduces cost and payload size while preserving semantic coverage for cues that change more slowly than the native frame rate.

2. Batch VLM inference (primary artifacts): Sampled frames are sent to Qwen2.5-VL-7B-Instruct using a chat-completions style endpoint [8]. The model is instructed to detect all visible people in each frame and produce one JSON object per batch containing per-frame detections, pixel-space bounding boxes, confidence, and prompt-conditioned attributes.

The system parses the model output as JSON and validates structure using a predefined schema (Pydantic) [9]. This step enforces field presence/types/ranges but does not guarantee semantic correctness (e.g., it cannot prove that a box tightly encloses a person).

A downstream annotator consumes the detections artifact and draws bounding boxes/labels onto the video to produce an annotated MP4. This is a visualization step; it should be treated as rendering model hypotheses, not ground truth.

The current prompting contract assigns person_id values uniquely within a frame (IDs restart from 0 per image); cross-frame identity tracking is not implemented. “Tracking” behavior in annotation logic should therefore be described as heuristic and potentially incorrect if it merges identities. In addition, schema validation is structural: bounding box coordinates may be non-negative and confidence may fall in [0, 1], while geometric constraints (e.g., x_max ≤ width, x_min < x_max) are requested by prompt but not guaranteed by validation. Finally, the interactive single-frame endpoint uses Llama-4-Scout for qualitative inspection; it returns free-form text rather than schema-enforced JSON, and should not be described as producing production-grade structured artifacts.

This project depends on one core idea: treat images like token sequences, then reuse the Transformer’s attention mechanism to integrate text instructions with visual evidence. The background below is restricted to concepts directly needed to understand the two chosen models and their system implications.

Transformers represent inputs as token sequences and apply self-attention to build contextual representations [2]. The key operation is scaled dot-product attention:

Here, Q, K, and V are learned projections of token embeddings and d is the key/query dimensionality. In multimodal settings, the “token sequence” may include both text tokens and visual tokens; attention then becomes the fusion mechanism that allows a generated output token to depend jointly on the prompt and the image.

Most VLMs convert an image into a sequence of embeddings using a vision backbone, often

ViT-style patch tokenization [3]. These visual tokens are then aligned to the language model’s embedding space and inserted into the context window so the decoder can attend across a unified sequence. This framing matters for localization-like tasks: a bounding box is ultimately a function of how well spatial information is preserved in the visual tokens and how reliably the decoder can translate that spatial evidence into pixel-coordinate outputs.

Instruction-tuned models are trained to follow prompts and produce controlled outputs [10]. In engineering pipelines, a common pattern is “structured generation”: prompts specify an explicit output contract (e.g., JSON) and downstream code validates structure [11]. This yields two important distinctions:

• Structural correctness (parsable JSON, required fields present, numeric ranges respected) can be enforced by schema validation [9].

• Semantic correctness (boxes correspond to people; attributes reflect visual evidence) cannot be proven by schema validation and remains model-dependent.

MoE architectures scale capacity using conditional computation: multiple expert sub-networks exist, but only a subset are activated per token [12]. A conceptual MoE layer can be expressed as:

where f i are experts and g i (x) are routing weights produced by a gating function. MoE is relevant here because Llama-4-Scout is documented as an MoE-based model in deployment references, positioning it as an efficient choice for interactive usage at its scale [5].

Qwen2.5-VL-7B-Instruct is an instruction-tuned vision-language model released by the Qwen team and distributed via a public model card [4]. It accepts multimodal inputs (text + image)

and generates text outputs that can be natural language or structured formats when prompted appropriately. Architecturally, it follows a widely used VLM pattern:

1. Vision encoder (image → visual tokens): An image is converted into a sequence of embeddings, commonly using ViT-style patch representations processed with self-attention [3]. The model card highlights support for dynamic resolution processing, which matters in practice when frames vary in resolution/aspect ratio [4].

Visual embeddings are mapped into a representation space that the language decoder can consume, using a projection/adapter and multimodal formatting conventions (e.g., special tokens and ordering) [4]. This is powerful (it unifies “detection + attribute reasoning” under one prompt) but also fragile:

the model is generating coordinates as tokens. As a result, even if the output is structurally correct JSON, boxes can be imprecise, off-by-constant, or inconsistent under occlusion/motion blur. In a defensible report, Qwen should therefore be described as a grounded generator that can be prompted to emit boxes, not as a detector that guarantees detector-grade localization.

Why Qwen is used for batch artifacts in the project. The system’s batch path needs an interface contract suitable for storage, downstream aggregation, and annotation. Qwen2.5-VL is a practical fit because instruction tuning supports compliance with strict formatting constraints (field names, coordinate format, confidence fields), enabling schema-oriented prompting and downstream structural validation [4,11]. In the BodyLanguageDetection repository [1], Qwen is prompted to detect all visible people per frame and return per-person bounding boxes and attributes (emotion by default). The architectural tradeoff is clear: one model can do perception + reasoning + structured output, but correctness must be treated probabilistically rather than guaranteed.

Meta-llama/Llama-4-Scout-17B-16E-Instruct is documented in deployment references as a multimodal, instruction-tuned model and is described as incorporating Mixture-of-Experts conditional computation [5,6]. Because full architectural details are not publicly disclosed, this discussion is limited to documented characteristics and standard multimodal Transformer patterns, avoiding claims that cannot be sourced.

typically encode an image into token-level representations and place those tokens into the context window so that attention can integrate visual evidence with the instruction prompt [2]. This architectural principle supports interactive tasks such as “explain what you see” or “describe visible cues” because generated text can be conditioned on both the prompt and image tokens.

MoE and interactive efficiency. The primary architectural reason to discuss Scout in this project is MoE. Under the MoE framing, only a subset of experts are activated per token, which can improve efficiency at a given capacity scale [12]. This makes Scout a plausible choice for interactive, developer-facing inspection where turnaround time matters.

Role in the project: interactive qualitative inspection. In the current implementation context [1], Scout is used for a single-frame endpoint intended for prompt iteration and qualitative debugging. The endpoint returns free-form text rather than schema-enforced JSON and does not serve as the structured extraction engine. This separation of roles is a system decision grounded in model behavior: structured generation is brittle to formatting drift and validation requirements, while free-form interactive responses are useful for rapid understanding and iteration.

Table 1: Comparison of selected models with respect to project requirements and architectural implications.

Strong fit: instruction-tuned VLM that can be prompted to emit strict JSON fields for batch artifacts [4,11].

Not used as a structured artifact generator in the current interactive endpoint; returns qualitative free-form text in typical usage context.

Batch extraction prioritizes machine-consumable artifacts; interactive inspection prioritizes human readability and iteration speed.

Feasible via generation: can output pixel-coordinate boxes as tokens when prompted; accuracy remains model-dependent [4].

Better suited to qualitative spatial descriptions; explicit box generation is not the focus of the interactive path.

Bounding-box overlays require per-frame box outputs, motivating Qwen in the batch path while keeping Scout for debugging.

Batch usage is higher overhead due to multi-frame processing and strict formatting constraints.

Documented MoE framing positions it as efficient for interactive usage [5,12].

The system separates roles to avoid coupling experimentation with the structured artifact interface.

Unified multimodal token sequence fused by attention [2,3].

Multimodal attention-based grounding in deployment descriptions; model-specific internals not asserted beyond documentation [5,6].

Both models satisfy multimodal reasoning; the differentiator is output contract strictness and operational role.

Can produce valid JSON with incorrect semantics; structured formatting can drift [13].

Free-form text avoids JSON parsing failures but is less directly consumable as an artifact.

Structured generation improves integration when it works; interactive free-form output improves developer velocity and interpretability.

This section documents only the implementation behaviors required to keep architectural claims honest and system interfaces defensible.

The batch path invokes Qwen2.5-VL through a hosted chat-completion style endpoint (e.g., via

Hugging Face inference providers / OpenAI-compatible interfaces) [8,14]. Frames are sampled and processed in small batches to control payload size and reduce service-limit failures.

The interactive endpoint invokes Llama-4-Scout on a single frame and returns a short free-form response. This endpoint should be described as a developer tool for qualitative inspection rather than a structured extraction component.

The prompt defines the output contract. In batch inference, the instruction requests:

• detection of all visible people in each frame,

• per-person pixel-space bounding boxes and confidence,

• and prompt-conditioned attributes stored in a flexible field (emotion by default).

Because attributes are prompt-conditioned, the system must not claim a fixed taxonomy unless it is explicitly enforced. For example, “emotion” labels may be expected in the default configuration, but additional keys in an analysis_result-style dictionary are fundamentally task-dependent and determined by user.

Downstream validation checks structural properties (field presence, types, basic numeric ranges) using a schema validator [9]. This improves robustness of parsing and downstream code. However, schema validation does not prove correctness of detections:

• A JSON object can be valid while boxes do not tightly match a person.

• Prompt-requested geometric constraints (e.g., box inside image bounds, x_min < x_max) may not be enforced unless explicitly validated.

A defensible system therefore treats schema validation as a necessary interface layer, not as correctness assurance.

In the current prompting contract, person_id values are frame-local (IDs restart per image).

Cross-frame identity tracking (re-identification, temporal matching, consistent IDs across time)

is not implemented and must not be implied. Any annotation logic that appears to “track” identities should be described as heuristic and potentially incorrect if it merges different people under the same ID.

Hosted inference introduces failure modes: invalid JSON, rate limits, and provider errors [8].

A robust report should acknowledge that batches may fail and that the system may continue without producing outputs for every sampled frame depending on error handling. This is not a minor detail: it affects whether the pipeline can be described as exhaustive versus best-effort.

This report adopted an architecture-led approach, using model design principles to motivate and justify system-level claims. Both selected models share a common multimodal foundation in which images are encoded as visual tokens and fused with instruction text through Transformer attention mechanisms [2,3]. Qwen2.5-VL-7B-Instruct follows the standard vision-encoder, multimodal-alignment, and Transformer-decoder paradigm and is therefore well suited for batch artifact generation, where compliance with strict JSON output formats enables reliable downstream parsing and visualization [4,11]. Llama-4-Scout-17B-16E-Instruct, documented as a Mixture-of-Experts multimodal model, is used to support efficient interactive analysis and prompt iteration through qualitative, free-form responses [5,12].

The primary outcome of this architecture-led perspective is clarity about system guarantees.

Within the implemented system context [1], schema validation enforces structural correctness and improves integration reliability, but it does not ensure semantic accuracy of detections or attributes.

In summary, the system design reflects a deliberate alignment between model capabilities, interface contracts, and enforced constraints. Making these boundaries explicit provides a sound basis for technical credibility and responsible deployment.