Counterfactual Neighborhood Risk Fields: Evidence-Grounded Selective Safety for Vision-Language Models
中文摘要
CNRF框架通过构建对比证据,提出了一种基于表征差异的视觉语言模型选择性安全防御方法,有效提升了风险评估的可信度。
English Summary
CNRF introduces an evidence-grounded framework using counterfactual pairs to improve selective safety and risk assessment interpretability for vision-language models.
arXiv:2610.20679v1 Announce Type: new Abstract: Representation-based safety defenses for vision-language models (VLMs) often learn to separate harmful from benign inputs, but separation alone does not establish what a risk score means or when it should be trusted. We introduce Counterfactual Neighborhood Risk Fields (CNRF), an evidence-grounded framework for selective VLM safety. CNRF constructs context-matched benign-harmful counterfactual pairs, uses their representation differences as capability-related risk directions, and estimates neighborhood support separately from risk. This separation prevents unfamiliarity from being conflated with harmfulness and enables selective intervention rather than blanket rejection. A victim-adaptive interface accommodates different safety observations across VLM architectures and routes selected requests for same-victim response review, which either releases acceptable candidates or produces constrained alternatives with reduced harmful capability. Across three VLMs and five multimodal jailbreak families, CNRF achieves 86.06-89.30% macro defense success rate while retaining substantial benign-task utility. Controlled experiments further show th…