SynthCorp: A Platform for Creating Benchmarks for Data Agents
中文摘要
SynthCorp通过模拟商业事件生成定制化数据,为数据智能体提供动态、可扩展的评估基准,解决现有基准易饱和的问题。
English Summary
SynthCorp generates customizable data estates and benchmarks by simulating business events, enabling more effective evaluation of LLM-driven data agents and overcoming saturation in existing testing frameworks.
arXiv:2611.14765v1 Announce Type: new Abstract: Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities. We present SynthCorp, a platform that creates customizable data estates and benchmarks by simulating how business events produce and modify corporate data over time. Each data estate combines relational databases, a document store, and a data catalog, paired with evaluation tasks and executable gold programs. First, given a scenario and target complexity dimensions, SynthCorp synthesizes reusable event types and compiles each into a program over predefined simulation primitives. Second, SynthCorp schedules event instances on a virtual calendar, interleaving routine and disruption events so that tables, documents, and catalog entries remain consistent. Third, SynthCorp executes the calendar to materialize the data estate and records every operation in a simulation log. Fourth, SynthCorp generates analytical tasks and derives verified gold programs from the log. We use SynthCorp to generate initial evaluation benchmarks comprising seven data estates (up to 1K operational tables) and 256 tasks across five domains. The data estates pass…