Counting the Bandages: On measuring the time that AI saves
中文摘要
这篇论文探讨了衡量AI节省时间的方法,提倡采用考虑模型错误率的“账簿法”,而非仅计算时长,并引入了“盈亏平衡缺陷”概念来评估其实际效益。
English Summary
The paper argues that AI time savings must account for counterfactual errors rather than simple clock time, proposing a "ledger view" and a "break-even defect" metric for assessment.
arXiv:2610.35942v1 Announce Type: new Abstract: I consider what it means for AI assistance to save time. I contrast a stopwatch view, on which time saved is whatever the clock records, with a ledger view, on which it is a counterfactual that has to be read next to the errors the work lets through, and I argue for the latter. The argument has four parts. I define a model-free pair of quantities, time saved and excess escaped defects, and one decision number built from them: the break-even defect cost. I give a baseline model of checking in which, until the model is good enough not to check, a calibrated worker's escaped-defect rate does not depend on how good the model is; checking time grows with the logarithm of the model's error rate; and over-trust buys minutes on the clock at a larger cost in total minutes. I show what randomized studies can and cannot identify, and I replicate METR's developer study from its public data: the same 246 issues support a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on somewhere between 3% and 82% of them. Finally, I restate the published record, with stated bounds in place of import…