RobinNet: A Multimodal Speech Emotion Recognition System With Speaker Recognition for Social Interactions
Yash Khurana, Swamita Gupta, R. Sathyaraj, S. P. Raja
Abstract
It is essential to understand the underlying emotions that are imparted through speech in order to study social communications as well as to generate seamless human–computer interactions. Speech emotion recognition (SER) is a considerably challenging task due to the lack of sufficient data and the complex interdependence of phrases with the context and emotion they imply. This article presents RobinNet: a RoBERTa-and Inception-ResNet-V2-based novel multimodal network for SER. The model employs transfer learning to build two unimodal systems for text and audio features and then incorporates them into a single classifier through Intermediate Fusion. This work has been created after carefully analyzing the performance of various top-performing unimodal systems and then utilizing a fine-tuned RoBERTa-based model to represent the textual features. Furthermore, we utilize an Inception-ResNetV2 pretrained network for Speaker Identification and employ transfer learning to train it for the task of emotion recognition through speech using spectrogram augmentation. The proposed multimodal system combines the two modalities through intermediate fusion and gives out a weighted accuracy (WA) of 72.8% when evaluated against the interactive emotional dyadic motion capture (IEMOCAP) dataset. Experimental results reveal that the proposed multimodal system outperforms state-of-the-art (SOTA) solutions on the benchmark datasets IEMOCAP, multimodal emotion lines dataset (MELD), and CMU-MOSEI. The proposed model utilizes intermediate fusion unlike any of its predecessors that perform late fusion after significant independent processing, thereby improving the overall artificial multimodal representations.