A hybrid LSTM-SNN approach for robust multimodal zero-shot learning
Yuejia Li, Zhe Yang, Haonan Zheng, Xiang Zhang
Abstract
Zero-shot learning (ZSL) has gained significant attention for its ability to identify previously unseen classes by leveraging features extracted from known classes, thus minimising the need for extensive training data. However, existing ZSL methods often fall short in accurately capturing temporal information in multimodal datasets, particularly in audio and video contexts, leading to suboptimal recognition performance. To address this challenge, we propose TempSimNet, a novel framework that combines long-short-term memory (LSTM) networks with spiking neural networks (SNN). LSTM excels at extracting long-term temporal dependencies, while SNN processes these features with high temporal precision through spike-based encoding. The integration of these two networks in TempSimNet enables effective temporal feature extraction and dynamic processing, enhancing the model's ability to recognise unseen classes in multimodal datasets. Experimental results demonstrate that TempSimNet achieves state-of-the-art performance across multiple benchmark datasets, significantly outperforming traditional ZSL approaches, particularly in generalised ZSL tasks.