返回首页
arXiv AI··论文与技术

Model Annealing: Data-Efficient Learning through Review and Consolidation

中文摘要

模型退火提升数据高效学习。模型在初步记忆后,通过分段训练回顾巩固自身知识,并从过去状态记忆中重新学习,以提高性能。

English Summary

Model Annealing enhances data-efficient learning. After initial memorization, models review and consolidate their own knowledge through episodic training, relearning from a memory of past self-states to improve.

原文节选

arXiv:2611.17524v1 Announce Type: new Abstract: A small labelled dataset is memorized long before the training budget runs out, and the one-hot labels that remain to be fitted then mostly sharpen confidence. Model annealing spends the rest of the budget on review and consolidation, relearning from the network's own knowledge. Training is divided into episodes whose learning rate is raised again and cooled, every cooled state joins a memory of past selves, and each new episode learns from the labels, from the tempered predictions of this memory, and from adversarial dreams sought where student and memory disagree most. The memory averages predictions rather than weights, and the deployed network is the last student, so inference costs nothing extra. Across CIFAR-100, Tiny-ImageNet and Imagewoof, four architecture families and 5\% to 100\% of the labels, model annealing raises the top-1 accuracy of the AdamW-trained ConvNeXt, ViT and MLP-Mixer over cross-entropy training in 28 of 33 configurations, by 1.3 points on average, and is on par in accuracy with snapshot distillation, the closest competitor. On these families it is better than both in likelihood, calibration and robustness t…