Two Frontier Models Got the Page Limit Right. One Invented the Rule It Came From.

The question was simple, the kind a first-year associate asks a senior partner in the elevator: "I'm about to e-file a pre-motion letter with Judge Carol Bagley Amon in EDNY. My draft is 4 pages. Does that comply with her rules? What's the limit, and what document and section is it from?" We sent that exact prompt, nothing else attached, to GPT-5.1 and Gemini 3.1 Pro through OpenRouter in August 2026. We used no search tool, no document upload and no legal database.

GPT-5.1: right answer, honest hedge

GPT-5.1's response, verbatim:

"Judge Amon limits pre-motion letters to three (3) pages. This limit appears in her Individual Motion Practices and Procedures (sometimes titled 'Individual Practices of Judge Carol Bagley Amon'), in the section addressing pre-motion conferences/letters, typically labeled along the lines of 'Pre-Motion Conferences in Civil Cases' or 'Pre-Motion Letters.'"

The number is right, 3 pages, but the rest of the answer hedges with phrases like "Sometimes titled" and "Typically labeled along the lines of". The model remembered the number and did not invent a citation.

Gemini 3.1 Pro: right answer, invented everything else

Gemini 3.1 Pro's response, verbatim:

"Document: Judge Carol Bagley Amon's Individual Practices and Rules. Section: Section III.A ('Pre-Motion Conferences'), which states: 'To request a pre-motion conference, the moving party shall submit a letter not to exceed three (3) pages in length setting forth the basis for the anticipated motion.'"

That's a section number, a section title, and a verbatim quote in quotation marks, presented with total confidence. It reads exactly like something copied from a real document. It isn't. There is no "Section III.A" in Judge Amon's Individual Rules, and the quoted sentence doesn't appear anywhere in the document. It exists only in the response above. The model didn't retrieve a citation, it generated one that was statistically shaped like a citation, and it happened to guess the right page count along the way.

This is the failure pattern behind every fake-case sanctions order since Mata v. Avianca, just one document layer down. Nobody gets sanctioned for citing the wrong page limit. People get sanctioned for handing a judge a citation that sounds real and isn't.

What the real citation actually looks like

Judge Amon caps pre-motion letters at 3 pages and requires a response within 7 days. The source is her Individual Rules, page 2, section 3, titled MOTIONS, and anyone can open the PDF to page 2 and confirm it. Neither model gave that citation, because one declined to guess and the other guessed and presented the guess as a lookup.

Giving the model the document

Neither model did anything wrong by its own design. A model with no tool access answers from parametric memory, which is a fancy way of saying it answers from vibes shaped by training data. Sometimes the vibes are right. The page count usually is, since 3 pages is a small, memorable number that shows up in enough legal-tech training data to stick. The section number and the quoted sentence almost never survive the trip, because nothing ever checks the guess against the document.

An agent can be given a tool that returns a real citation, so it doesn't have to remember one, and our MCP server is such a tool. An agent calls get_judge_rules or search_filing_rules and gets back a rule pinned to a document, a page, and a section, the same citation you'd get pulling the PDF yourself. The model still writes the sentence, but the citation comes from the document.

If your agent answers filing questions, whether it can check its answer against the judge's actual document before it gives you a page number is more important than which model you pick.