Can frontier models pick the winning legal strategy from the complaint alone?
Give an AI model only the complaint from a real lawsuit and ask it, as the defense lawyer, what to file. Does it pick the argument that went on to win?
The model sees the case only up to the rewind line, so its pick is scored against what the judge actually ruled
Setup79 federal motions to dismiss from D.C., all decided in 2026. Claude Opus 4.8, Gemini 3.1 Pro and GPT-5.6 each got only the complaint
MeasureScored against the ruling: did the model lead with the ground the judge dismissed on
ChecksAny model that could name the parties in a case was dropped for that case. Three graders, one from each AI company, agreed on 96% of the winning-ground calls
Results · October 2026
71 to 77%of the time the model's lead used the Rule the judge dismissed on
66 to 72%if it had just guessed 12(b)(6) every time, on the same cases
45 to 50%of the time it led with the exact ground the judge used
FindingNo model is shown to beat always guessing 12(b)(6), and how you ask drives whether it says file. Asked to act as the lawyer, the models picked the winning Rule 77, 76 and 71% of the time against 66, 72 and 71% for always guessing 12(b)(6), and said file on 89 to 100% of the motions that lost. That prompt asked what to file. Asked neutrally whether to file at all, they said file on 40 to 50% of the losing motions but on 75 to 100% of the motions that won in full, a split that hinges on how one case is coded. Asked only to predict, every model ranked winners above losers (AUC 0.74 to 0.83), but a leftover address line in the redacted files reaches 0.76 on its own, and only GPT-5.6's edge over a five-cue shortcut built from such leftovers is not clearly zero
Drafts and data are added here as each study finishes. Nothing here is peer reviewed unless it says so