Enhancing healthcare recommendation systems with multimodal LLMs-based MOE architecture

Abstract

With the increasing availability of multimodal data, many fields urgently require advanced architectures capable of effectively integrating these diverse data sources to address specific problems. This study proposes a hybrid recommendation model that combines the Mixture of Experts (MOE) framework with large language models to enhance the performance of recommendation systems in the healthcare domain. The MOE framework introduces multiple independent expert models (Experts) to select and activate only a part of the expert models for processing each time the input data is processed, thereby reducing the amount of calculation and improving the professionalism of the processing. In contrast, large language models excel in processing unstructured data and provide strong classification capabilities. In our experiments, we built a small dataset for recommending healthy food based on patient descriptions and evaluated the model's performance on several key metrics, including Precision, Recall, NDCG, and MAP@5. The experimental results show that the hybrid model outperforms the baseline models, which use MOE or large language models individually, in terms of both accuracy and personalized recommendation effectiveness. Several issues that warrant further exploration were identified during the experiments. Image data provided relatively limited improvement in the performance of the personalized recommendation system, particularly in addressing the cold start problem, Then, the issue of reclassification of images also affected the recommendation results, especially when dealing with low-quality images or changes in the appearance of items, leading to suboptimal performance. The findings provide valuable insights into the development of powerful, scalable, and high-performance recommendation systems, advancing the application of personalized recommendation technologies in real-world domains such as healthcare.

Topics & Concepts