返回首页
arXiv AI··论文与技术

More Programs or More Rolls? Separating Coverage from Specialization in LLM Harnesses

中文摘要

研究人员提出评估框架,区分大模型在任务专用化与重复执行中的表现,精准衡量提示词工具的效果。

English Summary

Researchers introduced a framework to distinguish between LLM task specialization and gains from repeated execution, offering more accurate evaluations of automated prompt engineering harnesses.

原文节选

arXiv:2609.35873v1 Announce Type: new Abstract: Automated generation of LLM harnesses promises to improve inference through task specialization. Yet additional answer coverage can arise from repeated execution of the same program, making specialization difficult to identify. We introduce a controlled evaluation that separates answer coverage, repeatable task advantages, and gains from pre-execution selection. On 386 MATH-500 tasks, we compare eight generated harnesses plus a baseline with nine byte-identical baseline copies, using three executions per member. Identical programs yield 2.16 percentage points of repeat-averaged oracle headroom. Generated programs exhibit substantially more repeatable score patterns, but these chiefly reveal persistent weaknesses: losses relative to the baseline persist across all three repeats on 100 tasks, while persistent wins occur on only one task and are sensitive to answer extraction. The frozen selector gains 0.00 percentage points, and both populations reach 98.70% oracle coverage at 27 harness executions. Stable complementarity remains unresolved at three repeats. Supporting BIRD traces locate failures in mechanism implementation, activation,…