DietAI24 as a framework for comprehensive nutrition estimation using multimodal large language models

Abstract

Accurate dietary assessment is essential for health research. While smartphone-based food image recognition offers a convenient alternative to traditional methods, existing computer vision approaches struggle with real-world food images and analyze only basic macronutrients, limiting their utility for comprehensive nutritional research. We developed DietAI24, a framework for automated nutrition estimation from food images that combines multimodal large language models (MLLMs) with Retrieval-Augmented Generation (RAG) technology to ground the MLLM’s visual recognition in authoritative nutrition databases rather than relying on the model’s internal knowledge. In our work, we used the Food and Nutrient Database for Dietary Studies (FNDDS) as the authoritative nutrition database. Through this approach, DietAI24 enables accurate nutrient estimation without extensive data collection or model training. DietAI24 significantly outperforms existing methods when evaluated against commercial platforms and computer vision baselines using the ASA24 and Nutrition5k datasets. Performance is measured through mean absolute error (MAE). DietAI24 achieves a 63% reduction in MAE for food weight estimation and four key nutrients and food components compared to existing methods when tested on real-world mixed dishes (p < 0.05). Notably, DietAI24 estimates 65 distinct nutrients and food components, far exceeding the basic macronutrient profiles of existing solutions. DietAI24 demonstrates that integrating MLLMs with RAG and standardized nutrition databases can substantially improve dietary assessment accuracy while enabling comprehensive nutrient analysis. This framework offers a scalable solution for nutrition research and clinical applications, potentially transforming large-scale epidemiological studies and personalized dietary interventions through more accurate and less burdensome dietary data collection. Taking photos of food to track nutrition is convenient, but current apps often guess wrong about what nutrients are in your meals. We created DietAI24, a system that combines artificial intelligence with a trusted nutrition database. When someone takes a food photo, our AI recognizes the foods and looks up their exact nutritional values instead of guessing. We tested DietAI24 against popular nutrition apps using thousands of food images. Our system was 63% more accurate and can identify 65 different nutrients, not just calories and protein, but also important micronutrients like vitamin D, iron, and folate that affect your health. This technology could help researchers better understand diet-related diseases and help doctors give personalized nutrition advice. It makes tracking what you eat easier and more reliable for everyone. Yan et al. present DietAI24, a unified framework that identifies foods, estimates portion sizes, and computes 65 nutrients from real-world food photos using multimodal large language models. This tool outperforms existing methods and standardizes outputs for downstream analysis, enabling efficient, reproducible dietary measurement.

Topics & Concepts