Fine-grained food image classification and recipe extraction using a customized deep neural network and NLP
A. Razia Sulthana, T. Tilford, Stoyan Stoyanov
Abstract
Global eating habits cause health issues leading people to mindful eating. This has directed attention to applying deep learning to food-related data. The proposed work develops a new framework integrating neural network and natural language processing for classification of food images and automated recipe extraction. It address the challenges of intra-class variability and inter-class similarity in food images that have received shallow attention in the literature. Firstly, a customized lightweight deep convolution neural network model, MResNet-50 for classifying food images is proposed. Secondly, automated ingredient processing and recipe extraction is done using natural language processing algorithms: Word2Vec and Transformers in conjunction. Thirdly, a representational semi-structured domain ontology is built to store the relationship between cuisine, food item, and ingredients. The accuracy of the proposed framework on the Food-101 and UECFOOD256 datasets is increased by 2.4% and 7.5%, respectively, outperforming existing models in literature such as DeepFood, CNN-Food, Wiser, and other pre-trained neural networks.