返回首页
arXiv AI··论文与技术

Representation Alignment from a Few Paired Data via Distilled Optimal Transport

中文摘要

本文提出一种基于蒸馏最优传输的对齐方法,仅需少量配对样本即可高效对齐两个预训练的单模态编码器。

English Summary

This paper proposes a representation alignment method using distilled optimal transport, aligning two pretrained unimodal encoders with only a few hundred paired examples.

原文节选

arXiv:2610.12781v1 Announce Type: new Abstract: Aligning two independently pretrained unimodal encoders typically demands millions of paired examples. In this paper, we study the regime where only $N \sim 100$--$500$ pairs are available, while unpaired data on either side is abundant. Our method expresses every representation by its similarities to the $N$ paired anchors, which places both modalities in a common coordinate system; matches the two point clouds with a single entropic optimal transport problem solved over the entire pool, whose marginal constraints counteract the hubness that defeats nearest-neighbor matching; and distills the resulting plan into two lightweight encoders, so that the correspondence becomes inductive. We additionally put this marginal constraint in relation with the one-step hub corrections of the bilingual lexicon induction literature. Pairing is required only for the anchors: the unpaired collections may be drawn independently from the two marginals, so that no item outside the anchor set needs to have both of its modalities observed. We evaluate on image-text and molecule-text alignment, where our method outperforms existing approaches at the same a…