Gradient-Aligned Pair Selection for Personalized Preference Optimization
中文摘要
该研究提出梯度对齐样本选择方法,通过优化偏好对选择来增强大语言模型的个性化对齐效果,改进了DPO框架中的启发式方案。
English Summary
This research introduces Gradient-Aligned Pair Selection to improve personalized LLM preference learning, replacing heuristic methods in Direct Preference Optimization (DPO) to achieve better user alignment.
arXiv:2610.00061v1 Announce Type: new Abstract: Personalizing large language models (LLMs) requires aligning generation behavior with user-specific preferences rather than aggregate quality. While Direct Preference Optimization (DPO) provides a stable framework for preference learning, its effectiveness in personalized settings critically depends on how preference pairs are selected. Existing approaches typically rely on heuristic criteria, such as likelihood-based extremes, which decouple optimization from explicit user utility and can lead to degraded personalization. We formalize personalized preference learning as a geometry-aligned optimization problem by analyzing the first-order interaction between gradients of expected user utility and DPO update directions. Our analysis reveals that, under off-policy sampling, the DPO update transitions from a purely error-corrective signal to a reinforcement-like update when preference margins are directionally aligned with utility gradients. This perspective exposes pair selection as a geometric decision that governs whether preference optimization advances or hinders personalization. Motivated by this insight, we propose GAP-DPO (Geomet…