Significant contributions include the development of the Weighted Group Synchrony algorithm, which enhances emotion classification by analyzing synchronization in physiological data, and the creation of the EmotiphAI platform and G-REx dataset, which enable synchronized real-time data collection and annotation, further boosting the accuracy and applicability of emotion recognition in real-world scenarios.
Physiological-based Group Emotion Recognition: Novel Methods and Real-World Applications
The thesis of Patrícia Bota, co-supervised by Pablo Cesar, focuses on improving emotion recognition systems through group dynamics.
Publication date
19 Sep 2024
Share this page
Emotions are a powerful phenomenon that influences the way we think and behave. Affective Computing is the field dedicated to the development of automatic emotion recognition systems. These systems can be used to improve the quality of life by the development of, for example, mental health applications, providing personalized services, or improving the quality of entertainment content, among others.
Patricia Bota, PhD student at Instituto Superior Técnico (IST) and CWI, supervised by Ana Fred (Instituto Telecomunicações – IT, IST), Hugo Plácido da Silva (IT, IST), and Pablo Cesar (CWI, TU Delft), during her PhD, began with an analysis of current emotion recognition technologies, reviewing and benchmarking the latest methods and features against public datasets. Emotion classification was assessed for arousal and valence using supervised learning, with a focus on the advantages of feature fusion and multi-modality, which proved effective in achieving results comparable to or better than existing studies.
Additionally, group dynamics were examined through the introduction of a novel Weighted Group Synchrony method, which reflects synchronization in physiological data for emotion recognition. Results indicate that integrating group data enhances classification accuracy, with significant improvements in arousal and valence detection.
Analyzing the available devices for physiological data collection,a limitation was observed in existing technologies for centralized group settings. To address this gap, the EmotiphAI platform was developed, enabling synchronous real-time data acquisition with up to 10 devices at 60 Hz. Taking that data needs to be annotated to be used in artificial intelligence algorithms, the EmotiphAI Annotator was introduced. The platform utilizes an innovative retrospective annotation system that selects relevant physiological segments for efficient and accurate emotion annotation, demonstrating high usability and consistency among raters.
Lastly, leveraging the EmotiphAI physiological data and annotation tools, the G-REx dataset was created. This new dataset, comprising over 380 hours of data from more than 190 subjects during naturalistic movie sessions, validates the replicability of the EmotiphAI setup in real-life scenarios, and paves the way for advancements in Affective Computing.
CWI’s Distributed and Interactive Systems research group focuses on facilitating and improving the way people use interactive systems and how people communicate with each other. We combine data science with a strong human-centric, empirical approach to understand the experience of users. This enables us to design and develop next generation intelligent and empathic systems. We base our results on realistic testing grounds and data sets, and embrace areas such as ubiquitous computing, human-centered multimedia systems, and languages.
More information
PhD Defense Patrícia Bota: Wednesday 25 September 2024, 10h-12.00h at Instituto Superior Técnico, Lisbon
Promotors: Prof. Ana Fred (IT, IST), Prof. Hugo Silva (IT, IST), and Pablo Cesar (CWI, TU Delft)
Relevant publications
P. Bota, J. Brito, A. Fred, P. Cesar, and H. Silva. 2024. A real-world dataset of group emotion experiences based on physiological data." Scientific Data 11.1. https://doi.org/10.1038/s41597-023-02905-6
P. Bota, P. Cesar, A. Fred and H. P. da Silva. 2024. Exploring Retrospective Annotation in Long-videos for Emotion Recognition. IEEE Transactions on Affective Computing. https://doi.org/10.1109/TAFFC.2024.3359706
P. Bota, T. Zhang, A. El Ali, A. Fred, H. P. da Silva and P. Cesar. 2023. Group synchrony for emotion recognition using physiological signals. IEEE Transactions on Affective Computing 14.4 (2023): 2614-2625. https://doi.org/10.1109/TAFFC.2023.3265433
Related News
- https://www.cwi.nl/en/stories/only-a-heartbeat-away/
- https://www.cwi.nl/en/news/fine-grained-temporal-emotion-recognition-in-video/
- https://www.cwi.nl/en/news/monitoring-the-engagement-of-groups-by-using-physiological-sensors/
- https://www.cwi.nl/en/news/smart-textiles-and-the-internet-of-things/
- https://www.cwi.nl/en/news/cwi-dis-group-in-red-bull-doc-this-is-the-future/
