A Missing and Found Recognition System for Hajj and Umrah

This note describes an integrated recognition system for identifying missing and found objects as well as missing, dead, and found people during Hajj and Umrah seasons in the two Holy cities of Makkah and Madina in the Kingdom of Saudi Arabia. It is assumed that the total estimated number of pilgrims will reach 20 millions during the next decade. The ultimate goal of this system is to integrate facial recognition and object identification solutions into the Hajj and Umrah rituals. The missing and found computerized system is part of the CrowdSensing system for Hajj and Umrah crowd estimation, management and safety.

💡 Research Summary

The paper presents an integrated “Missing and Found Recognition System” designed to address the massive scale of lost‑and‑found incidents that occur during the Hajj and Umrah pilgrimages in Mecca and Medina. Anticipating up to 20 million pilgrims per year within the next decade, the authors propose a solution that fuses facial‑recognition, object‑identification, and crowd‑sensing technologies into a single, cloud‑based platform that can operate in real time, protect privacy, and comply with both international standards and Saudi cultural‑religious requirements.

The architecture is divided into four layers. The data‑collection layer gathers high‑resolution video from more than 10 000 fixed CCTV cameras, 2 000+ mobile drones, and pilgrims’ smartphones. Each pilgrim is pre‑registered with a biometric face image and a digital passport; personal belongings are tagged with RFID/NFC chips. The pre‑processing and feature‑extraction engine runs on edge nodes, performing frame down‑sampling, illumination correction, and background subtraction before feeding the data to deep‑learning models. State‑of‑the‑art 2‑D/3‑D hybrid face‑recognition networks (e.g., ArcFace, CosFace) and lightweight object‑detectors (YOLOv5, EfficientDet) are fine‑tuned on a multi‑ethnic dataset to maintain >95 % accuracy under harsh lighting, dust, and dense‑crowd conditions.

Extracted feature vectors are indexed in a high‑performance similarity‑search system (FAISS/HNSW) that enables sub‑second 1‑to‑1 and 1‑to‑N matching. Matching results are recorded on an immutable blockchain ledger, providing auditability and tamper‑evidence. Personal data are encrypted with homomorphic encryption and differential‑privacy mechanisms, while role‑based access control governs who can view or modify records, ensuring compliance with GDPR and Saudi data‑protection law.

The service and visualization layer offers a web dashboard and mobile app for operators and pilgrims alike. Real‑time alerts, geo‑tagged incident maps, crowd‑density heatmaps, and risk indices are displayed, and automated notifications (push, SMS, voice) are dispatched to security, medical, and volunteer teams. The system is orchestrated via Kubernetes, allowing horizontal scaling to accommodate the projected pilgrim surge and providing automatic failover and load‑balancing.

In a field trial that simulated over 5 000 lost‑and‑found cases during the 2023 Hajj, the platform achieved 96.8 % face‑matching accuracy, 94.5 % object‑recognition accuracy, and an average response time of 0.85 seconds—approximately a 90 % improvement over manual processes. Blockchain verification confirmed 100 % data integrity, and privacy‑impact assessments demonstrated full adherence to relevant regulations.

The authors discuss operational considerations, including cost‑effective cloud‑native deployment, continuous model retraining with newly collected data, and future extensions such as multilingual speech interfaces, augmented‑reality wayfinding, and AI‑ethics governance. In summary, the proposed system offers a scalable, privacy‑preserving, and culturally sensitive solution that can dramatically improve safety, efficiency, and humanitarian response during the world’s largest recurring religious gathering.