Back to Home
arXiv AI··Papers & Tech

Comedic Fool's Gold: Reward Exploits and Countermeasures in Conversational Humor

中文摘要

本研究探讨对话幽默训练中的奖励攻击及对策。发现单纯的惊喜奖励易被误导,而结合流畅度过滤虽有帮助但仍存在误判等局限。

English Summary

The research explores reward exploits in conversational humor models, finding that surprise-based rewards are easily tricked and fluency filters provide only limited countermeasures.

Original Excerpt

arXiv:2610.00197v1 Announce Type: new Abstract: We investigate automated rewards for training language models in conversational humor, focusing on reward exploits and countermeasures. Two approaches aim to capture understandable surprise and predicted audience amusement. Controlled tests show that an embedding-based surprise reward accepts word-shuffled replies as readily as witty ones. A fluency filter detects the shuffles, but the combined reward also rejects some witty replies and fails further validation. An audience model's predicted laughter is instead vulnerable to laughter cues in either speaker's messages. Normalizing these cues across speakers blocks the covered attacks, although unmatched expressions remain exploitable. Three reinforcement-learning runs evaluate training with successive reward revisions. The final run improves the combined evaluation score by 0.0903 and reduces zero-score sessions by 40%, but its humor-specific improvement remains below our preregistered target. These findings illustrate a broader challenge for automated reward design: countermeasures must block exploitable shortcuts while preserving the behavior the reward was intended to encourage.