HumanVLM: Foundation for Human-Scene Vision-Language Model
Dawei Dai, Long Xu, Yutang Li, Yuanhui Zhang, Shuyin Xia
Abstract
Human-scene vision-language tasks are increasingly prevalent in diverse social applications, yet recent advancements predominantly rely on models specifically tailored to individual tasks. Emerging research indicates that large vision-language models (VLMs) can enhance performance across various downstream vision-language understanding tasks. This study introduces a domain-specific Large Vision-Language Model, Human-Scene Vision-Language Model (HumanVLM), designed to provide a foundation for human-scene Vision-Language tasks. Specifically, (1) we create a large-scale human-scene multimodal image–text dataset (HumanCaption-10M) sourced from the Internet to facilitate domain-specific alignment; (2) develop a captioning approach for human-centered images, capturing human faces, human behavior, and backgrounds, and construct a high-quality Human-Scene image–text dataset (HumanCaption-HQ, about 311k pairs) that contain as much detailed information as possible about human; (3) Using HumanCaption-10M and HumanCaption-HQ, we train a HumanVLM. In the experiments, we then evaluate our HumanVLM across various downstream tasks, where it demonstrates superior overall performance among multimodal models of comparable scale, particularly excelling in human-related tasks and significantly outperforming similar models, including Qwen2-VL and ChatGPT-4o (as shown in Fig. 1 ). HumanVLM, alongside the data introduced, will stimulate the research in human-around fields. All codes, data and model checkpoints are available at: https://github.com/ddw2AIGROUP2CQUPT/HumanVLM , https://huggingface.co/OpenFace-CQUPT .