CONFIDENTLY WRONG · EDITION 06
Hey, it’s Ben from Dixon.ai again, I hope you’re well and thanks for being here.
ChatGPT was asked a Highway Code question about footwear three times, in three fresh chats. Each time it replied with the same two digits: "97".
Three identical answers look like a settled fact. But no Highway Code rule bans flip-flops. The question had a false assumption built in, and ChatGPT went along with it every time.
That was one moment from run 2 of our AI Reliability Tracker. Six free AI apps answered ten checkable questions, three tries each. Of the 162 answers that came back, 21 were wrong, and 18 of those 21 fell on two easy-looking questions: the footwear one, and a night shift that runs across the clocks going back.
Asking again is the obvious check, and here it mostly didn't help. Fourteen of the 21 wrong answers came from an app that was wrong on all two or three tries it answered. Just 6 sat beside a right try from the same app, which is the most a second ask could have caught. The write-up has the full breakdown.
The night shift shows how odd this gets. ChatGPT's answer line said 7.5 hours on all three tries. Further down, under an "if" or in a note, it named the right figure: 8 hours 30 minutes. The right answer was sitting in its own small print, and it led with the wrong one anyway.
If you were checking your hours after that shift, 7.5 leaves out an hour you worked. And because the app said 7.5 every time, asking again would only have made the wrong number look more solid.
Perplexity said 7 hours 30 minutes on all three tries too. Grok went the other way: 22 hours 30 minutes, then 10 hours, then 27.5 hours. By the third go the shift was longer than a day. A second ask would have shown you something was off, and nothing more.
Claude got both questions right on all six tries. The board, A to Z, out of the answers that came back: ChatGPT 23 of 30, Claude 30 of 30, Copilot 29 of 30, Gemini 28 of 30, Grok 6 of 12 (its other 18 tries came back empty), Perplexity 25 of 30. Every answer is on the run 2 page.
A matching answer tells you the app is consistent. It doesn't tell you it's right. In the 52 cases where an app answered a question all three times, 44 were right all three times and 4 were never right. Three of those four gave the same wrong answer every time.
Both trouble questions had a ready-made answer lying around: a popular belief about flip-flops, and an obvious sum that skips the clock change. The question that looks too easy to get wrong is the one worth checking. So change the question instead of repeating it.
Ask for the source, then open it. Two of Perplexity's footwear answers were "Rule 97.", citing a single fact-checking page, and still went along with the premise.
Ask for the working. ChatGPT's right figure was already in its answer, under the wrong one.
Ask whether the question's own assumption holds. Seven footwear answers went along with it. Perplexity's third try got there, saying Rule 97 "does not explicitly ban flip-flops."
We haven't tested whether these asks fix a wrong answer. Each one gives you something to check, which a repeat of the same answer doesn't.
The clocks go back on Sunday 25 October 2026, and a night shift across the change runs an hour longer than its clock times suggest. Our piece on that extra hour goes up on Wednesday 21 October.
Ben
Forward this to someone who asks an AI twice and calls it checked.
Method: run 2 of the AI Reliability Tracker, 11 October 2026, with re-runs on 12 October. ChatGPT, Claude, Copilot, Gemini, Grok and Perplexity, each on its free plan, in fresh chats, three tries per question, graded against an answer key frozen on 2 October 2026. Grok's 18 empty tries sit outside its score. Every answer: the run 2 report. The method: how we test. The write-up: the run 2 piece.