Back to Home
arXiv AI··Papers & Tech

CARAT: Do Materials LLMs Reason or Recite?

中文摘要

CARAT框架测试材料LLM是推理晶体结构还是背诵事实。它通过匹配视图、证据注入等方法,区分模型理解与记忆,解决现有评估局限性。

English Summary

CARAT is a new framework testing if materials LLMs truly reason about crystal structures or just recite facts. It uses methods like matched views and evidence injection to distinguish reasoning from memorization, addressing limitations of accuracy metrics.

Original Excerpt

arXiv:2609.38340v1 Announce Type: new Abstract: When a materials LLM answers a question about crystal structure, does it reason from the structure or copy an answer already printed in its input? Accuracy cannot tell: a structural description often prints the very field it is scored against. CARAT holds question and gold answer fixed across eight matched views, names each structural relation separately in GraphSpace, and adds matched fine-tuning, answer masking, evidence injection, paired inference, and a rule that can withhold claims. First, on the benchmark's hardest families the grounded view is worth 17.3 points over formula inputs. Second, we turn that scrutiny on ourselves. GraphSpace beats a plain periodic graph by 19.3 points, but that margin is two effects at once: where the plain rendering carries everything the question needs it is 1.96 points, and where it omits those fields entirely, 46.7 points. The headline mostly measures what the baseline lacked, not how evidence is presented. Third, we attack our own benchmark. A rule that skips the link and reads the list directly answers four of seven hardened families, so we rebuilt it until eleven such shortcuts sat near chance…