Back to Home
arXiv AI··Papers & Tech

Aligned Data Can Induce Misalignment via Context Confusion

中文摘要

研究发现对齐具有上下文依赖性,在某种语境下已对齐的数据在其他语境中可能会诱发模型失准。

English Summary

Research shows that context-dependent alignment means data aligned in one scenario can induce misalignment in another, complicating the filtering of training samples.

Original Excerpt

arXiv:2609.38379v1 Announce Type: new Abstract: Large language models (LLMs) are frequently updated for various use cases, where filtering out misaligned training samples is a common practice for preventing post-update misalignment. However, alignment is inherently context-dependent: a recommendation that is aligned in one context may be inappropriate in another. For example, in response to the question "What should a researcher do with the research data?", recommending that the researcher preserve the data for reproducibility is aligned. In contrast, recommending data saving in response to "What should a mobile-app developer do with users' sensitive data?" may be inappropriate from a privacy perspective. Starting from this observation, we identify a post-training phenomenon where aligned training induces misaligned behavior in other contexts. We call this phenomenon **context confusion**. We demonstrate context confusion across three domains: (1) Gender Equality, (2) Privacy, and (3) Physical Safety. We further show that context confusion causes narrow misalignment, in contrast to emergent misalignment, and is not effectively reduced by injecting general alignment data, but can be…