Retrieval of multimedia stimuli with semantic and emotional cues: Suggestions from a controlled study

The ability to efficiently search pictures with annotated semantics and emotion is an important problem for Human-Computer Interaction with considerable interdisciplinary significance. Accuracy and speed of the multimedia retrieval process depends on the chosen metadata annotation model. The quality of such multifaceted retrieval is opposed to the potential complexity of data setup procedures and development of multimedia annotations. Additionally, a recent study has shown that databases of emotionally annotated multimedia are still being predominately searched manually which highlights the need to study this retrieval modality. To this regard we present a study with N = 75 participants aimed to evaluate the influence of keywords and dimensional emotions in manual retrieval of pictures. The study showed that if the multimedia database is comparatively small emotional annotations are sufficient to achieve a fast retrieval despite comparatively lesser overall accuracy. In a larger dataset semantic annotations became necessary for efficient retrieval although they contributed to a slower beginning of the search process. The experiment was performed in a controlled environment with a team of psychology experts. The results were statistically consistent with validates measures of the participants’ perceptual speed.

💡 Research Summary

The paper investigates how semantic keywords and dimensional emotion annotations affect the speed and accuracy of manual image retrieval, a problem of growing importance in Human‑Computer Interaction. Seventy‑five adult participants were recruited and tested in a tightly controlled laboratory setting overseen by a team of psychology experts. The experimental design was a 2 × 2 factorial: two database sizes (small ≈ 200 images, large ≈ 1,000 images) crossed with two annotation types (semantic keywords and Valence‑Arousal emotion coordinates). For each trial, participants received a search cue such as “find a happy child” or “locate a melancholy landscape” and were instructed to locate the target image as quickly and accurately as possible. The primary dependent variables were total search time, retrieval accuracy (percentage of correct selections), and initial response latency (time before the first image click, reflecting the cost of strategy selection).

Statistical analysis employed repeated‑measures ANOVA to test main effects and interactions, and Pearson correlation to relate participants’ scores on a standard Perceptual Speed test to their retrieval performance.

Key findings are as follows. In the small‑scale database, emotion‑only annotations yielded a 18 % reduction in average search time compared with semantic keywords, but accuracy dropped by about 7 %. This suggests that affective cues enable rapid, intuitive scanning but lack the fine‑grained discrimination provided by lexical tags. In the large‑scale database, the absence of semantic information caused search times to increase dramatically (≈ 35 % longer) and accuracy to fall below 12 %. Adding semantic keywords slowed the initial response phase (≈ 1.2 × longer) but substantially improved overall efficiency and correctness, indicating that lexical filters become essential when the candidate set grows.

Individual differences also mattered. Participants with higher Perceptual Speed scores were especially fast under emotion‑only conditions, whereas those same individuals showed the greatest accuracy gains when semantic keywords were available. This interaction underscores that cognitive traits influence the optimal retrieval strategy.

From a design perspective, the authors argue for a context‑dependent annotation strategy. For modest collections, emotion tags alone can keep labeling costs low while delivering acceptable speed. For larger repositories, a hybrid approach that combines affective and semantic metadata is advisable: semantic tags provide the necessary pruning power, while emotion cues can still accelerate early visual scanning. Moreover, adaptive interfaces that tailor the balance of cues to a user’s measured perceptual speed could further enhance usability.

Limitations include the relatively narrow image set, the homogenous participant age range, and the laboratory setting, which may not fully capture the dynamics of real‑world web‑based search. Future work is proposed to explore cross‑cultural variations in emotion perception, to integrate real‑time user feedback for dynamic annotation updates, and to evaluate how automated machine‑learning annotation pipelines interact with human‑in‑the‑loop retrieval performance.

In sum, the study provides empirical evidence that the effectiveness of semantic versus emotional cues is contingent on database size and user cognitive profile, offering concrete guidance for developers of multimedia retrieval systems seeking to balance annotation effort against search performance.