Advancing Multimodal Visual Analytics for Disaster Management with LLM-Enhanced CLIP

Abstract

Abstract

The increasing volume of visual and textual data shared on social media during disasters offers valuable opportunities for improving situational awareness, yet poses significant challenges for reliable automated analysis. This study introduces LLM2CLIP, a vision–language framework designed to enhance multimodal disaster classification by integrating visual cues with linguistically grounded textual representations. Built upon the CrisisMMD benchmark, LLM2CLIP replaces CLIP’s original text encoder with a large language model fine-tuned for image–text alignment, improving cross-modal semantic understanding while preserving computational efficiency through transfer learning. The framework is evaluated across three key crisis-related tasks—informativeness detection, humanitarian categorization, and damage severity assessment—and consistently outperforms existing unimodal and multimodal baselines. The findings highlight the potential of LLM-based textual embeddings to strengthen multimodal coherence, robustness, and generalization in noisy real-world data. By advancing the integration of vision and language in disaster analytics, this work contributes to the development of scalable, robust, and data-driven systems that support timely decision-making in crisis management.