Real movement versus measurement noise

Here is an experiment anyone can run in ten minutes. Take a query you care about. Ask it. Write down the sources cited. Close the tab, open a new one, ask exactly the same question. You will not get the same list. That's not a bug you've stumbled on, and it isn't personalization. It's the normal operating behaviour of every system in this category, and its size has now been measured well enough that we can put numbers on it — which changes what any before-and-after report is worth.

How much it moves

Parse looked at prompts that returned at least twelve source-citing answers over a month, covering 16,143 ChatGPT prompts and 15,805 Google AI Overview prompts across 693,509 answers. Repeat answers to the same prompt shared 21.2% of their cited sources on ChatGPT and 31.5% on AI Overviews. Narrowing to a seven-day window barely helped, lifting those to 26.7% and 36.8%.

ChatGPT21.2%

Sources shared between repeat answers

Sources that turn over

81.3 distinct domains drawn across ~22 runs of one prompt.

AI Overviews31.5%

Sources shared between repeat answers

Sources that turn over

A typical prompt has 1.43 domains that show up in 80%+ of answers.

Semrush found the same thing from a different direction. Taking 1,500 US desktop keywords that produced an AI Overview on at least twenty of October's thirty-one days, not one keyword held its URL set for the month. Ninety-six per cent saw a domain change. Ninety-one per cent of the URLs studied dropped out at some point, and only 43% of those ever came back. A given URL held its slot for an average of 3.87 consecutive days. Authoritas, using an edit-distance volatility index across 11,203 keywords sampled on three dates, put AI Overview citation volatility at 0.68 to 0.73 against 0.49 to 0.55 for organic rankings — roughly 1.4 times as unstable. Their read: over two to three months, expect around 70% of the pages appearing in an AI Overview to change.

Why it drifts, which is less mysterious than it sounds

The usual explanation is that models are probabilistic and you should expect variation. That explanation is wrong, or at least it's answering a different question, because this variation shows up at temperature zero, where the model is supposed to pick the highest-probability token every time. Thinking Machines Lab published the actual mechanism in September 2025. Sample a thousand completions from a large model at temperature zero — same prompt, same settings — and you get eighty unique outputs. All thousand are identical for the first 102 tokens. At token 103 they split: 992 continue one way, 8 go another, and from there they diverge. The cause isn't randomness in the model. It's that the kernels doing the arithmetic aren't batch-invariant. Operations like RMSNorm, matrix multiplication and attention change the order in which they sum floating-point numbers depending on how many requests are being processed together. Server load varies. Batch size varies with it. Your answer changes because of who else was querying at the same moment. The proof is that it's fixable. With batch-invariant kernels enabled, all one thousand completions came back identical — at a cost of roughly 60% more processing time. Nobody serving at production scale is paying that, which is precisely why every public endpoint you measure against is non-deterministic by construction. Peer-reviewed work backs this up at the outcome level. A study presented at Eval4NLP in December 2025 ran five API-based models across eight tasks, ten times each, with deterministic settings configured. Accuracy varied by up to 15% between runs of the same setup. As the authors put it, no model consistently delivers the same outputs or accuracies, regardless of task.

This problem has a name and it is older than any of this

Measurement science worked all of this out long before anyone was tracking AI citations. The international vocabulary of metrology defines repeatability as precision under conditions that include the same procedure, same operators, same system, same conditions, and replicate measurements over a short period. It defines random measurement error as the component that varies unpredictably across replicate measurements. Reliability, in that framing, is a ratio: true variance divided by true variance plus error variance. If the error term is large, the ratio collapses, and no amount of dashboard polish recovers it. In fields that take this seriously, an intraclass correlation below 0.5 is classified as poor reliability and results built on it are not reported. Nobody in AI visibility reporting publishes a reliability figure. It would be an uncomfortable number.

What a three-point gain is actually worth

The most rigorous statistical treatment of this specific problem is a 2026 arXiv paper by Ronald Sielinski, which sampled three engines across three consumer topics both daily over nine days and at ten-minute intervals, then bootstrapped confidence intervals around the resulting citation shares. One worked example is worth the whole paper. On SearchGPT for running gear at n=200, one domain showed a citation share of about 9.5% with a 95% confidence interval spanning 5.5% to 12.5%. A second domain sat at about 6.0%, interval 4.0% to 8.0%. The apparent 3.5-point gap between them is entirely contained within the sampling noise. The two are statistically tied. The paper generalizes it: overlapping intervals are the norm rather than the exception whenever domains appear to differ in citation share by less than five to seven percentage points. Which means most of the week-over-week movement in most AI visibility dashboards is the instrument breathing.

How many runs you actually need

The same paper gives sample-size guidance, targeting a confidence interval width of 0.05 for citation share. Gemini reached that at roughly 30 runs on one topic and 40 to 50 on others. Perplexity needed 90 to 100. SearchGPT never converged cleanly, and the author warns explicitly that the apparent flattening as n approaches 200 is a bootstrap artifact rather than evidence that the platform stabilizes.

SparkToro arrived at a similar answer by an entirely different route. Six hundred volunteers ran twelve prompts through ChatGPT, Claude and Google's AI surfaces 2,961 times in late 2025. The finding: there's less than a one-in-a-hundred chance that two responses to the same prompt will contain the same list of brands, and closer to one in a thousand that they'll be in the same order. Their recommendation was to run each prompt at least sixty to a hundred times and average. Their conclusion about what you can and can't sell from that data is worth quoting directly: visibility percentage across dozens to hundreds of prompts run multiple times is a reasonable metric, and any tool that reports a ranking position in AI is not.

What to do with this

Three things follow, and none of them are complicated. Establish your noise floor before you report a change. Run your own prompt set repeatedly against an unchanged page and see how much the number moves on its own. That figure is the threshold below which you have nothing to report. Separate the parts of your measurement that drift from the parts that don't. Anything read off your own HTML — whether a crawler can reach the page, whether a clean answer can be parsed from it, whether the passage survives extraction — is deterministic and should return the same value every time. Anything read off a live answer surface is sampled and needs a run count attached. Blending them into one score hides which half moved. Report intervals, not points. If a stakeholder asks whether the number went up, the honest answer is often "by less than the measurement can resolve." That is a real answer, and it's better than the alternative, which is redirecting a quarter of work on the strength of a rounding error.

Sources

Related reading

• A single content score can't tell you what to fix — a trend line is only worth trusting if the number underneath it is decomposable.
• Publishing velocity: what the evidence actually says — a worked example of a metric nobody stress-tested.
• Not every prompt sends an AI to the web — one reason answer surfaces look noisier than they are.
• How it works — scheduled scans and the what-changed report after every run.