Journal:Thirty-Ninth AAAI Conference on Artificial Intelligence (AAAI), CCF-A
Abstract:Supervised multimodal classification has been proven to outperform unimodal classification in the image-text domain. However, this task highly relies on abundant labeled data. To perform multimodal classification in data-insufficient scenarios, in this study, we explore semi-supervised multimodal classification (SSMC) that only requires a small amount of labeled data and plenty of unlabeled data. Specifically, we first design baseline SSMC models by combining known semi-supervised pseudo-labeling methods with the two most commonly used modal fusion strategies, i.e. feature-level fusion and label-level aggregation. Based on our investigation and empirical study of the baselines, we discover two complementarities that may benefit SSMC if properly exploited: the predictions from different modalities (modal complementarity) and modal fusion strategies for pseudo-labeling (strategic complementarity). Therefore, we propose a learning from Modal and Strategic Complementarity (MSC) framework for SSMC. Concretely, to exploit modal complementarity, we propose to learn reliability weights to weigh the predictions from different modalities and refine the fusion scores. To learn from strategic complementarity, we introduce a dual KL divergence loss to guide the balance of quantity and quality of pseudo-labeled data selection. Extensive empirical studies demonstrate the effectiveness of the proposed framework.
Co-author:Junchi Chen,Richong Zhang,Junfan Chen
Indexed by:国际学术会议
Page Number:15812--15820
Translation or Not:no
Date of Publication:2025-01-01