Emotion categorization from facial expressions: A review of datasets, methods, and research directions

Abstract

Emotion classification plays a crucial role in the domain of human–computer interaction, as it holds substantial significance in effective communication. However, the task of emotion classification presents notable challenges primarily due to the subjective and multifaceted nature of emotions, encompassing varying cultural and individual interpretations. Numerous methodologies have been employed to address the complexities associated with facial emotion classification. Notably, contemporary techniques such as those that consider emotion-relevant features (e.g. mouth, eyes) have demonstrated a certain degree of success in comparison to alternative approaches (e.g. the widely employed end-to-end deep networks), which consider all pixel information in an image with equal importance. Nonetheless, a comprehensive set of guidelines evaluating not only the strengths, but also the weaknesses of deep networks remains absent. These approaches are vulnerable to even minor alterations in images, regardless of their relevance to emotion classification, e.g. a change in the foreground color of the image can reduce the accuracy of these methods by a significant amount. This results in a fragmented field, ultimately missing opportunities to enhance performance for emerging real-world applications. Therefore to address this issue, this paper offers a comprehensive review of state-of-the-art datasets and research, providing insights into different techniques and their unique contributions to emotion categorization. We also thoroughly analyze current challenges and issues, laying the groundwork for promising future research directions. Our review reveals the significance of considering contemporary methods, particularly in contrast to widely used deep networks, and their potential to enhance performance and application success. • The paper provides an in-depth exploration of datasets for facial emotion categorization. • It highlights deep learning’s limitations, revealing drawbacks in emotion categorization. • The model’s vulnerability to changes irrelevant to emotion is discussed. • The need for robust methods accounting for real-world variability is emphasized. • The paper provides a roadmap for enhancing emotion categorization methods.

Topics & Concepts