Why one AI answer is not a measurement
By Rastock AEO teamPublished 6 min read
One AI answer is a single draw from a system that gives different answers to the same question, so it cannot tell you how an engine usually describes your business. A measurement needs a fixed set of questions, repeated runs, stored answers and a like-for-like comparison over time.
Someone on your team asks ChatGPT for the best provider in your field, sees your name, and sends a screenshot round the office. A week later a customer asks the same thing and gets a list without you. Neither person is wrong. They each saw one answer out of many possible ones.
Why the same question gets different answers
A search results page is fairly stable. An AI answer is written fresh each time, and several things can shift it.
- Built-in randomness. Language models choose words with a degree of chance. Anthropic's documentation says that even at the most conservative setting, identical inputs can still produce different outputs.
- Wording. "Best accountant in Leeds" and "who should I use for my small business taxes in Leeds" are the same need to a person, but can lead to different lists.
- Live search. When an engine searches the web, the answer depends on which pages it retrieved that time. Google describes its AI features running a set of related searches behind a single question, so the pool of source pages can differ from one question to the next.
- Location. OpenAI says ChatGPT search uses general location information based on IP address to improve results. A question asked in another city or country may be answered differently.
- Model version. Providers replace and update their models. An answer from the spring may come from a different model than an answer today, and free and paid plans may not use the same one.
- Personal context. Some assistants can draw on earlier conversations or saved preferences. An account that has discussed your company before is not a neutral observer.
What a single screenshot proves
A screenshot proves that this answer was possible, once, for that account, in that place, with that wording. That is real information. If the answer states a wrong fact about you, it is worth acting on straight away.
What it cannot tell you is how often that answer appears. A good screenshot does not mean you are usually recommended. A bad one does not mean you are usually missing. And two screenshots taken before and after a website change cannot show that the change caused the difference, because the answer might have changed anyway.
Sampling: the basic method
Measuring AI answers borrows a plain idea from surveys. You cannot see every answer an engine gives, so you collect a sample in a consistent way and describe what you saw.
- 1Fix the question set. Write down the questions real customers ask, in their words. Include questions that name you ("is [business] any good") and questions that do not ("best [service] in [town]"). Do not change the list casually, or you lose the ability to compare.
- 2Repeat each question. Run every question several times per engine, on different days. One run tells you what can happen. Repeated runs start to show what tends to happen.
- 3Keep the conditions steady. Use a clean session with no chat history or memory, note the engine and model shown, and note the location and language. If you change one of these, record it.
- 4Store the full answer. Keep the complete text, the list of cited sources and the date and time. A tick in a spreadsheet saying "mentioned" loses the detail you will want later, such as what exactly was said about your prices.
- 5Count in simple terms. "Mentioned in 7 of 20 runs" is honest and easy to understand. Avoid turning a small sample into a precise-looking score.
Mentioned is not the same as cited
Two different things can happen in an answer, and they should be recorded separately.
| Mentioned | Cited | |
|---|---|---|
| What it means | Your business name appears in the answer text | A page is linked as a source for the answer |
| Whose page | No page needed | Your own site, or someone else's page that talks about you |
| What it tells you | The engine associates you with this question | Which pages the engine actually read |
| What to check | Is the description accurate and fair? | Are the facts on that page correct and current? |
You can be mentioned with no link at all. Your site can be cited in an answer that recommends a competitor. And a directory page about you can be cited while your own site is ignored, which tells you where the engine is getting its facts.
Citation counts alone need care too. Bing's own AI citation report warns that its numbers do not indicate the placement, ranking or role of a page within an answer.
Comparing before and after
The point of measuring is usually to learn whether a change helped. A fair comparison has a few rules.
- Take a baseline before you change anything, using the full question set and repeated runs.
- After the change, wait. Engines need time to recrawl pages, and no engine promises a schedule.
- Run the same questions, in the same engines, under the same conditions, the same number of times.
- Compare like with like: this question in this engine before, against this question in this engine after.
- Read the stored answers, not only the counts. A mention that moved from "expensive" to "mid-priced" is a change that a count would miss.
Being honest about noise
Even with a careful method, results wobble. If you were mentioned in 6 of 20 runs last month and 8 of 20 this month, that may be ordinary variation and not progress. Small moves in small samples should be reported as "no clear change".
A few habits keep reporting honest:
- Say how many questions and runs sit behind every number.
- Look for changes that hold across several rounds, not one.
- Do not claim your edit caused a change when a model update happened in the same period. Say both happened.
- Remember the limit of any outside view. Google states that no third-party tool has access to its internal ranking or AI systems. Sampling shows what engines said to you, not why.
- Keep the raw answers so anyone can check the summary against them.
This is the approach Rastock AEO is being built around: fixed questions, repeated runs, stored answers, and mentions separated from citations. That measurement is in build and not yet live; only early-access accounts exist. The method does not depend on any tool, though. A question list, a clean browser session and a spreadsheet are enough to begin.
Questions and answers
- How many times should I ask the same question?
- There is no magic number, and this article does not offer one. More than once is the minimum, several runs on different days are better, and whatever number you pick should stay the same in every round.
- Is a screenshot of an AI answer useless?
- No. It proves that answer was possible and can reveal a wrong fact worth fixing. It just cannot show how often the answer appears or whether a change you made caused it.
- What is the difference between being mentioned and being cited?
- Mentioned means your name appears in the answer text. Cited means a page, yours or a third party's, is linked as a source. They can happen independently, so record them separately.
- Why did the answer change when I did nothing?
- Answers vary through built-in randomness, different retrieved pages, location, model updates and personal context. A change without any action from you is normal and is the reason single answers should not be over-read.
Sources
See what AI says about your business.
Get early accessMore articles
What is answer engine optimization (AEO)?
A plain guide to answer engine optimization: what answer engines are, how AEO relates to SEO, what you can influence, what you cannot, and first steps.
robots.txt, llms.txt and AI crawlers: what a small site should set up
Which AI crawlers visit your site, the official user-agent names to allow or block in robots.txt, what llms.txt really is, and the JavaScript-only page risk.