Attention Monitoring and Hazard Assessment with Bio-Sensing and Vision: Empirical Analysis Utilizing CNNs on the KITTI Dataset

Assessing the driver’s attention and detecting various hazardous and non-hazardous events during a drive are critical for driver’s safety. Attention monitoring in driving scenarios has mostly been carried out using vision (camera-based) modality by tracking the driver’s gaze and facial expressions. It is only recently that bio-sensing modalities such as Electroencephalogram (EEG) are being explored. But, there is another open problem which has not been explored sufficiently yet in this paradigm. This is the detection of specific events, hazardous and non-hazardous, during driving that affects the driver’s mental and physiological states. The other challenge in evaluating multi-modal sensory applications is the absence of very large scale EEG data because of the various limitations of using EEG in the real world. In this paper, we use both of the above sensor modalities and compare them against the two tasks of assessing the driver’s attention and detecting hazardous vs. non-hazardous driving events. We collect user data on twelve subjects and show how in the absence of very large-scale datasets, we can still use pre-trained deep learning convolution networks to extract meaningful features from both of the above modalities. We used the publicly available KITTI dataset for evaluating our platform and to compare it with previous studies. Finally, we show that the results presented in this paper surpass the previous benchmark set up in the above driver awareness-related applications.

💡 Research Summary

**
The paper presents a multimodal framework for assessing driver attention and distinguishing hazardous from non‑hazardous driving events by jointly exploiting electroencephalogram (EEG) recordings and forward‑facing video captured from a vehicle. Twelve participants watched fifteen video clips drawn from the publicly available KITTI dataset while seated in a driving simulator. During the sessions, a compact 14‑channel Emotiv EPOC headset recorded brain activity at 128 Hz, and a webcam captured the driver’s face. All streams were synchronized using Lab Streaming Layer.

EEG preprocessing involved artifact removal with EEGLAB’s ASR pipeline, band‑pass filtering (4–45 Hz), and two complementary feature extraction strategies. First, mutual information (conditional entropy) was computed for every pair of the 14 electrodes, yielding 91 pairwise features that capture inter‑regional brain interactions. Second, power spectral density (PSD) was calculated for the theta (4–7 Hz), alpha (7–13 Hz), and beta (13–30 Hz) bands, averaged across each trial, and visualized as 2‑D topographic heatmaps. The three band‑specific heatmaps were encoded as RGB channels, resized to 224 × 224 × 3, and fed into a pre‑trained VGG‑16 network. The activations from the penultimate fully‑connected layer (4,096 dimensions) were extracted as deep EEG features. These deep features were concatenated with the mutual‑information vector, providing a rich representation despite the modest dataset size.

Facial analysis followed a two‑stage pipeline. A Haar‑Cascade detector located the face, after which the Chehra algorithm supplied 49 facial landmarks. From these landmarks, 30 geometric descriptors (distances and angles) were computed; their mean, 95th percentile, and standard deviation across frames produced 90 statistical features per trial. In parallel, each face crop was passed through the VGG‑Faces network (pre‑trained on 2.6 M face images), and the same 4,096‑dimensional deep descriptor was extracted and summarized across time (mean, 95th percentile, std). The geometric and deep descriptors were merged to form the visual feature set.

To capture temporal dynamics, the authors generated multiple successive EEG‑PSD images and facial feature windows within each trial. After extracting VGG‑16 (EEG) or VGG‑Faces (face) descriptors for each window, principal component analysis reduced the dimensionality to 60 components. The resulting 60 × N matrices (N = number of windows) were fed into a Long Short‑Term Memory (LSTM) network, enabling the model to learn trends over the trial. Because KITTI video lengths varied, this trend‑based LSTM was applied only to the hazard‑detection task, where trial durations were standardized.

Two classification tasks were evaluated: (1) driver attention level (high vs. low) and (2) hazardous vs. non‑hazardous situation detection. Conventional classifiers (SVM, Random Forest) and the LSTM‑based temporal model were trained on EEG‑only, vision‑only, and fused feature sets. Compared with a prior KITTI‑based hazard detection benchmark (≈78 % accuracy), the proposed system achieved higher performance: EEG‑only ≈84 % accuracy, vision‑only ≈86 %, and multimodal fusion ≈90 % (F1 scores showed similar gains). Notably, EEG contributed discriminative power in short (1–2 s) intervals where facial expressions remained unchanged, highlighting its value for rapid cognitive state detection.

The authors acknowledge several limitations. The sample size (12 subjects, predominantly in their twenties) restricts generalizability, and the low‑density Emotiv headset offers limited spatial resolution compared to clinical EEG systems. The fusion strategy is a simple concatenation without learned weighting, and statistical significance testing of the reported improvements is absent. Moreover, the LSTM trend analysis could not be applied to the attention‑assessment task due to variable video lengths.

In conclusion, the study demonstrates that pre‑trained convolutional networks can be repurposed to extract informative representations from both EEG topographies and facial imagery, enabling effective driver state monitoring even with modest data volumes. Future work should explore larger, more diverse participant cohorts, higher‑density EEG hardware, and more sophisticated multimodal fusion architectures (e.g., attention‑based or graph‑neural networks) to move toward real‑time, in‑vehicle deployment.