Evaluating and improving AI agents for sequencing protocol understanding
中文摘要
LibStructBench评估科学AI代理理解测序方案及重建分子结构的能力,旨在提升其对复杂实验流程的分析精度。
English Summary
LibStructBench evaluates AI agents on their ability to reconstruct sequencing protocols and molecular workflows, addressing gaps in current scientific benchmarking for complex experimental analysis.
arXiv:2610.37851v1 Announce Type: new Abstract: Existing scientific-agent benchmarks rarely evaluate whether agents can reconstruct the exact molecular structures and generation workflows specified by sequencing protocols, a capability critical for analyzing customized sequencing assays. We introduce LibStructBench, a document-grounded benchmark for reconstructing sequencing-library structures and molecular workflows. Our experiments show that frontier agents often produce approximately correct reconstructions but remain unreliable at exact reconstruction, particularly in molecular-state representation and workflow connectivity. Switching the underlying language model produced larger average performance differences than switching agent harnesses, with a statistically significant difference only for molecular-state reconstruction. Additional knowledge improved oligo-family recovery in a small pilot without improving mean molecular-workflow scores. Autonomous accumulation of procedural rules and cross-protocol examples showed non-monotonic validation performance and no mean held-out molecular-workflow improvement in the tested trajectory. These findings highlight the need for more co…