LifeGraph 4 - Lifelog Retrieval using Multimodal Knowledge Graphs and Vision-Language Models
Luca Rossetto, Athina Kyriakou, Svenja Lange, Florian Ruosch, Ruijie Wang, Kathrin Wardatzky, Abraham Bernstein
Abstract
In the scope of the 7th Lifelog Search Challenge (LSC'24), we present the 4th iteration of LifeGraph, a multimodal knowledge-graph approach with data augmentations using Vision-Language Models (VLM). We extend the LifeGraph model presented in former LSC challenges by event-based clustering using temporal and spatial relations as well as information extracted from descriptions of Lifelog image captions produced by VLMs.
Topics & Concepts
LifelogComputer scienceArtificial intelligenceInformation retrievalNatural language processingKnowledge graphWorld Wide WebHuman–computer interactionTopic ModelingMultimodal Machine Learning ApplicationsAdvanced Graph Neural Networks