Back to Home
arXiv AI··Papers & Tech

Cross-Modal Latent Influence: Patch-Level Training Sample Attribution for Vision-Language Models

中文摘要

介绍MOSAIC方法,为视觉-语言模型提供补丁级训练样本归因,使预测影响力可追溯至具体的跨模态空间证据。

English Summary

MOSAIC enables patch-level training sample attribution for VLMs, allowing predictions to be traced back to specific multimodal spatial evidence.

Original Excerpt

arXiv:2610.14190v1 Announce Type: new Abstract: Explaining a vision--language model (VLM) prediction requires tracing training-example influence to the multimodal evidence that mediates it, yet existing attribution methods largely operate only at the sample level. Recent latent-mediated influence methods provide feature-level attribution in language models, but lack the multimodal and spatial structure needed to ground influence in VLMs. We introduce MOSAIC (Multimodal Orthogonalized Sparse Autoencoders for Influence Characterization). It learns a shared sparse image--text latent space whose concepts are 1) sparse within each modality, 2) supported on contiguous image regions, 3) aligned to the phrases that describe them, and 4) approximately decorrelated. It then applies EK-FAC to decompose training-example influence through these concepts and localize their contributions to specific text phrases and image regions. We evaluate MOSAIC on Qwen and Gemma across captioning, chest X-ray understanding, and visual question answering. The learned features are more localized and coherent without degrading task behavior, and ablating influence-ranked latents changes model outputs more than …