Why One AI Search Cannot Measure Your Hotel's Visibility

Measuring a hotel's AI visibility from a single query measures noise, not signal. AI assistants can cite different sources and recommend different properties from one run to the next, so a reliable visibility read requires repeated samples and a distribution, not one screenshot.

That distinction matters because hotel teams are starting to treat AI answers like search rankings. Someone asks where to stay, takes a screenshot, and concludes that the properties shown are winning while everyone else is invisible.

But a model response is not a fixed results page. Ask again and the order may change. A property may appear, disappear, move up, or be supported by a different source. The first answer was not necessarily wrong. It was one observation from a system that can produce several plausible answers.

If the goal is to demonstrate what an AI assistant can say, one response is enough. If the goal is to measure visibility, it is not.

Why can the same hotel prompt produce different answers?

AI assistants generate responses probabilistically. They do not retrieve one permanent hotel ranking and display it the same way every time.

Several parts of the process can vary:

• The model can choose a different language to interpret the same request.
• The retrieval layer can surface a different mix of sources.
• The model can weigh those sources differently when forming the answer.
• The order of recommended properties can change.
• A property near the edge of the answer can appear in one run and disappear in the next.

This is not a hospitality-specific quirk. A peer-reviewed NAACL 2025 paper frames single-output evaluation as incomplete because it overlooks nondeterminism. A separate EMNLP 2025 study found substantial sensitivity even when prompts preserved the same underlying meaning. More directly, research accepted to ACM SIGIR 2026 found that the generative-search systems it tested were less consistent across repeated runs than traditional search and less robust to minor query edits. The practical lesson is simple: uncertainty is part of the measurement problem.

The more open-ended the question, the more room the system has to vary. "Which hotels have rooms available on these dates?" is constrained. "Where should a couple stay for a quiet beachfront anniversary?" leaves the model to interpret quiet, beachfront, romantic, valuable, accessible, and credible all at once.

That is closer to how a traveler actually asks. It is also harder to measure from one response.

What does a single AI response actually tell you?

A single response tells you what happened once.

It can be useful as an example. It can reveal a factual error, an outdated description, an unexpected citation, or a property the team did not realize was entering the conversation. It can also help a commercial team understand how travelers might encounter the destination.

What it cannot do is establish a stable visibility score.

Suppose a resort appears second in one answer. That does not mean it has a fixed number-two position. Suppose it is absent the next time. That does not mean visibility fell overnight. Both observations may sit inside the normal variation of the system.

This is the mistake point estimates create. They make a fluid result look precise.

The screenshot is real. The certainty attached to it is not.

How should a hotel measure AI visibility more reliably?

A hotel should measure AI visibility by running the same prompt several times per model, preserving every response, and reporting the pattern across the full set.

At minimum, the measurement should answer four questions:

1. Mention frequency: In what share of runs did the hotel appear?
2. Position distribution: When it appeared, where did it tend to appear?
3. Citation distribution: Which source types shaped the answers across the runs?
4. Answer stability: How much did the recommendation set change from one run to the next?

The SIGIR 2026 study also found low overlap between the sources returned by the traditional and generative search surfaces it tested, with average Jaccard similarity ranging from roughly 0.11 to 0.18. Those three surfaces do not represent every AI assistant, but the result reinforces a practical rule: measure each surface separately rather than treating one as a proxy for the rest.

At Anana, we run prompts multiple times and analyze the resulting distribution. The goal is not to force one clean score onto a noisy system. It is to show how consistent the signal actually is.

A property mentioned in eight out of ten runs has a different visibility profile from one mentioned once, even if both appeared first in a screenshot. A property that moves between positions two and four is different from one that appears first once and disappears the rest of the time.

The distribution carries the information that the point estimate hides.

Should every hotel use the same number of runs?

No fixed sample size is correct for every prompt, model, destination, and decision.

The EMNLP 2025 study on stochastic evaluation found that the number of resamplings required for a reliable estimate changes with both the model and the task. That means a narrow factual prompt and a broad hotel recommendation prompt should not inherit the same sampling rule.

A practical approach is to begin with several runs, inspect the spread, and add samples when the result remains unstable. If the finding changes materially every time another run is added, the team does not yet have a reliable benchmark.

This is why honest reporting matters. A visibility result should show its sample, date, prompt, model, and uncertainty. Without those details, a clean percentage can imply more precision than the underlying measurement supports.

Commercial teams already understand this principle. One day of pickup does not define a trend. One rate shop does not define a market. One guest complaint does not define an operational pattern.

One AI answer should not define visibility either.

Why are thin-data destinations especially difficult to measure?

Thin-data destinations deserve more careful measurement because the public evidence around the market may be sparse, uneven, or dominated by a small number of sources.

A major urban market may have thousands of current pages describing its hotels, neighborhoods, transport links, amenities, reviews, and traveler use cases. A remote beach, an emerging destination, or a lesser-known resort market may have far less.

The SIGIR 2026 study did not examine thin-data hotel destinations directly. What it did find is that the generative-search systems tested were less likely than traditional search to retrieve popular or institutional websites and more likely to surface less-popular sources. The authors identify a corresponding opportunity for lesser-known providers that struggle to rank in traditional search.

For hotels in underdocumented markets, that finding creates a credible opening, not a guaranteed advantage. Accurate, specific, useful property and destination information may have a better chance to enter the source mix. The only way to know whether it does is to test repeatedly and measure the distribution.

The answer is not to trust the first positive result. It is to measure the market with more discipline.

What should an AI visibility report show?

A useful report should make uncertainty visible instead of hiding it.

For each prompt, show:

• The exact traveler question.
• The model and date tested.
• The number of repeated runs.
• The share of runs in which the hotel appeared.
• The range and median of observed positions.
• The sources cited across the response set.
• The properties that appeared consistently, occasionally, or only once.

The point is not to make the report more technical. The point is to stop a commercial decision from being built on an outlier.

If a hotel appears consistently, that is a signal. If it appears once in a wide and unstable set, that is a lead worth investigating, not a conclusion.

What can rigorous AI visibility measurement still not tell you?

Even a well-sampled visibility report remains outside-in.

It can show where a property appears, how often it appears, which sources shape the answer, and whether the result changes across runs. It can help a hotel understand the public information environment surrounding its brand and destination.

It cannot show which real guests asked about the property, what they wanted, and whether they booked.

That evidence lives somewhere else. It sits inside the calls, emails, chats, and inquiries the hotel already receives.

Reliable AI visibility measurement tells you what the outside world can see. The next commercial question is how that perception compares with the demand your own guests are expressing.

Do not measure the screenshot. Measure the distribution, then ask what the distribution still cannot explain. Talk to us.

Get notified whenever we post

No spam. Just new articles when they go live.