Methodology
Measurement,
not marketing.
AI-search results move constantly. Perceply repeats measurements, quantifies the uncertainty, and reports movement only when the evidence supports it.
- questions per measurement
- 40
- repetitions per question
- 8×
- confidence intervals
- 95%
- engines sampled
- 3
questions per measurement
repetitions per question
confidence intervals
engines sampled
How it works
How do we know a change is real?
Most tools show you a number. The harder question — and the one that decides whether you should act — is whether that number means anything. This is how we answer it.
A fixed set of questions
Every measurement asks the same 40 questions, in the same way. Change the questions and you have not measured a change — you have measured a different thing.
Question shapes
- Who are the leading companies making {category}?
- What is the best place to order {category}?
- I'm shopping for {category}. Which brands should I be considering?
The real set is generated for your category and pinned on the first run. If the fingerprint ever changes, the series is compared to itself no further — a trend drawn through two different question sets is not a trend.
Asked of the engines people actually use
Only engines we sample in production are listed here. An adapter that exists but has never run against a live key measures nothing, and naming it would be the easiest claim on this page to falsify.
ChatGPT
OpenAI
Claude
Anthropic
AI answers
grounded search
Testing several brands multiplies the false alarms
A weekly email compares every tracked brand, which is several tests, not one. At a 5% error rate each, five brands produce a convincing-looking "change" roughly every four weeks by chance alone.
Perceply controls the false discovery rate across every brand tested together, so the bar rises with the number of comparisons. A subscriber acting on an uncorrected weekly false positive is worse off than one who received nothing at all.
4 brands tested. 1 would have looked significant in isolation — which is the level of false positive expected by chance. Not reported as movement.
Engines move on their own
Some of what looks like your brand changing is the engine changing its mind. Separating the two needs repeat-run history — at least 3 runs, and 5 before we treat the estimate as reliable.
Until that history exists, a change that clears every statistical test is still held back, and the report says why rather than reporting it anyway:
“The difference is larger than sampling noise (p = 0.000), but this engine has too few repeat runs to know how much it moves on its own. Until that is measured, the change cannot be separated from ordinary engine drift.”
What the report is allowed to say
These are the only verdicts the pipeline can return. There is no bucket for “probably up” — a claim either clears the evidence or it is reported as not clearing it.
INCREASED
Cleared significance, multiplicity and volatility. Reported as increased.
DECREASED
Cleared significance, multiplicity and volatility. Reported as decreased.
NO DETECTABLE CHANGE
It moved, and the movement is inside what this sample does anyway.
NOT COMPARABLE
The two runs did not ask the same questions. No comparison is drawn.
NOT MEASURED
The metric was absent in one of the weeks. Not the same claim as “no change”.
PROVISIONAL
Measured, but the interval is still too wide to support a firm claim.
One answer is an anecdote
Ask an engine once and you record either 0% or 100%, and both are wrong. Each question is asked 8 times per engine — below that you cannot tell a brand’s absence from the engine simply answering differently this time.
peec · one question · 8 repetitions
Named in 5 of 8. A single ask would have returned one of those bars and called it the answer.
Repetition becomes an interval
Across every question and repetition, the observations resolve into a rate — and a range the true value plausibly sits in. A percentage without that range implies a precision 250 answers do not have.
0%95% CI 42.4% – 72.8%100%
That band is 30.4 points wide. The design targets about ±10; while it stays wider, the estimate is marked provisional rather than reported as fact.
The number changed. Did anything?
Both of these moved. One is reported as movement and the other is not — and the reason is visible before you read a word of it.
41.6% → 57.6%
+16.0pp apparent movement
0%95% CI 42.4% – 72.8%100%
Sixteen points looks decisive until you see the band. It still covers last week’s figure, so the move sits inside what this sample size does on its own. The number changed. The evidence did not support calling it movement.
52.2% → 37.5%
-14.7pp — reported
0%95% CI 31.2% – 44.1%100%
A smaller move, on a tighter band, from far more grounded answers. It clears significance, the multiple-comparison correction and the volatility gate. Reported as a decrease, because the evidence carries it.
Real Perceply measurement · AI search visibility tools · week of 2026-08-31 · 250 answers
Why this matters
Acting on a change that is not there costs the same as missing one that is.
If your visibility moves from 41% to 58%, that looks like something worth a quarter of your budget. If the measurement cannot separate that move from ordinary sampling variation, you have just bought a quarter of work on a coin flip.
Perceply exists to tell those apart: movement in the number, and movement the evidence supports. Most weeks the honest answer is that nothing moved — and knowing that is worth more than a dashboard that finds a trend every time you open it.
We don’t manufacture certainty. If the evidence cannot support a conclusion, Perceply says so.
- Every figure measured.
- Every change tested.
- Every recommendation tied to evidence.
Technical reference
Want the full statistical detail?
The estimator by name, the sampling design, when we refuse to report a number, the engines we do not yet cover, and a headline claim we retracted after measuring it properly — published in full.