Back to Home
arXiv AI··Papers & Tech

SlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code Repositories

中文摘要

SlackBench评估AI智能体在真实代码库协作项目中的表现,弥补了现有基准测试缺乏复杂多方交互及动态协作场景的不足。

English Summary

SlackBench evaluates AI agents on collaborative projects using real code repositories, addressing gaps in current benchmarks regarding complex multi-party interactions and dynamic workplace task environments.

Original Excerpt

arXiv:2611.05512v1 Announce Type: new Abstract: AI Agents are becoming digital coworkers that can browse the web, write code, run experiments, and assist humans over extended horizons. Yet existing benchmarks fall short of real knowledge work: many rely on synthetic data or isolated tasks, missing the complex, multi-party dynamics of collaborative projects, where critical information is scattered across conversations, code, and evolving decisions. Even simple workplace questions may require locating relevant channels and direct messages, recovering context from long discussions, inspecting repository state, reconciling stale or conflicting evidence, and synthesizing across sources. We introduce SlackBench, a benchmark for evaluating agents on realistic research and engineering workflows that require joint reasoning over workplace communication and evolving repository state. Grounded in privacy-preserving reenactments of real academic research projects, SlackBench provides reproducible environments with realistic multi-party dialogue, evolving decisions, and corresponding code changes grounded in GitHub repositories and Google Docs. It contains 140 queries spanning heterogeneous-sou…