Back to the research area

Mercury: a framework for Driver Monitoring and Human Car Interaction

The deep learning-based models and frameworks investigated at AImageLab have been exploited to implement a framework to monitor the driver attention level and to allow the interaction between the driver and the car through the Natural User Interfaces paradigm.
The driver attention is a concept that is difficult to define in a univocal way and, also, it has different nuances: for this reason, we focus on the concept of attention understood as the driver's level of fatigue, computed through the perclos measure, and the driver fine and coarse gaze estimation.



The whole architecture of the system is represented in the following figure.
The system has been set up in a modular way, since it is able to work only with RGB data, depth data or both. 




Acquisition Module

  • Microsoft Kinect One 


Head Extraction

  • RGB Images: Viola & Jones algorithm provided by the dLib libraries
  • Depth Maps: a combination of 


Image Analysis

  • Head Pose Estimation
  • Facial Landmark Localization
  • Eye State Analysis


Driver Monitoring

  • Perclos
  • Coarse Gaze Estimation


​Human Car Interaction


1 Borghi, Guido; Fabbri, Matteo; Vezzani, Roberto; Calderara, Simone; Cucchiara, Rita "Face-from-Depth for Head Pose Estimation on Depth Images" IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, pp. 1 -1 , 2018
DOI: 10.1109/TPAMI.2018.2885472 Journal
2 Borghi, Guido; Venturelli, Marco; Vezzani, Roberto; Cucchiara, Rita "POSEidon: Face-from-Depth for Driver Pose Estimation" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, Hawaii, July, 22-25, 2017, 2017 Conference
3 Borghi, Guido; Gasparini, Riccardo; Vezzani, Roberto; Cucchiara, Rita "Embedded Recurrent Network for Head Pose Estimation in Car" Proceedings of the 2017 IEEE Intelligent Vehicles Symposium, Redondo Beach CA, USA, June 11-14, 2017 Conference
4 Ballotta, Diego; Borghi, Guido; Vezzani, Roberto; Cucchiara, Rita "Head Detection with Depth Images in the Wild" Proceedings of the 13th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, Funchal, Portugal, 27-29 January 2018, 2017 Conference
5 Frigieri, Elia; Borghi, Guido; Vezzani, Roberto; Cucchiara, Rita "Fast and Accurate Facial Landmark Localization in Depth Images for In-car Applications" Proceedings of the 19th International Conference on Image Analysis and Processing, Catania, 11-15 september 2017, 2017 Conference
6 Venturelli, Marco; Borghi, Guido; Vezzani, Roberto; Cucchiara, Rita "From Depth Data to Head Pose Estimation: a Siamese approach" Proceedings of the 12th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISAPP), Porto, Portugal, 27 february - 1 march, 2017, 2016 Conference
7 Venturelli, Marco; Borghi, Guido; Vezzani, Roberto; Cucchiara, Rita "Deep Head Pose Estimation from Depth Data for In-car Automotive Applications" Proceedings of the 2nd International Workshop on Understanding Human Activities through 3D Sensors, Cancun (Mexico), Dec 4 , 2016, 2016 Conference

Video Demo

Research Activity Info