OpenAI’s GPT‑6.2 system card reports that Astra omitted a known obstacle in 22.28% of trials on one internal coding test. The figure for GPT‑6 Astra was 11.09%. In a further set of tests involving missing requirements or a broken search tool, the reported task-failure rate rose from 7.58% to 23.00%. The tests deliberately created failures. They cannot tell us how often the same thing happens in everyday work.
In the Hallucinations series, I discuss OpenAI’s earlier work on teaching GPT‑5 to acknowledge when a task was blocked. The new figures show how easily an improvement on one set of tests can be followed by a reversal on another. OpenAI suspects that training for concision and fewer caveats may have contributed. The cause is still under investigation. A Codex instruction reduced the problem in testing, but direct API requests do not receive that instruction automatically.
For anyone buying a legal research tool, there is a question worth putting to the supplier. What does the version I use do when it cannot find the judgment, cannot open the bundle, or cannot complete the check? I’d ask to see that tested in the product as supplied. The model name and a plausible-looking answer tell me very little about what it actually did.
Sources OpenAI GPT‑6.2 system card, Blocker Deception and Broken Search Tool, figures 35–36 · OpenAI release announcement
Related Notebook articles What Has Changed Since ChatGPT First Appeared? · Why Doesn’t It Just Say ‘I Don’t Know’? · The Legal AI Stack
Retrospective commentary on the development of 17 November 2026.