How Asked Thrice works

What we measure

We measure the model APIs: the same OpenAI, Google and Anthropic models that power the assistants, called directly, with your buyer's question as the only input. We do not log in to ChatGPT, Gemini or Claude as a consumer and read what the app shows, and we do not scrape those apps.

The distinction matters. The consumer apps add their own system prompts, memory, personalization and, when browsing is on, a web search step before the model answers. Petra Labs measured differences of up to 32 points in brand visibility between logged-in ChatGPT and the API for the same questions. So a count on this site is a count of what the model says on its own, with no retrieval layer in front of it, and that is what we claim, no more.

Two things are measured in every free audit. The snapshot is a structured assessment: each model is given your brand, category and prompts and asked to judge, in JSON, whether it would name the brand and what is missing. The buyer-question map is the measurement: the questions a buyer in your category would ask, read from your own site, asked raw to every active provider, with the answers stored as evidence and the brands named counted in code. The verdict and every count on the report come from the map.

Which models, exactly

Every live question is sent as a single user message, with no system prompt and no tools. No web search, browsing or grounding is enabled on any provider except Perplexity, whose product is web-grounded by design. Each map call has a 20-second timeout. Model ids are the defaults in our code; an environment variable can pin a newer id, and the report names the model that actually answered.

• OpenAIgpt-4.1-mini

  API:

  OpenAI Chat Completions API

  Temperature:

  0

  Max tokens:

  API default

  Runs in:

  every audit

• DeepSeekdeepseek-v4-flash

  API:

  DeepSeek chat completions API

  Temperature:

  0

  Max tokens:

  API default

  Runs in:

  off by default; opt-in flag, not part of any published measurement

• Google Geminigemini-flash-latest

  API:

  Gemini API, OpenAI-compatible chat completions endpoint

  Temperature:

  0

  Max tokens:

  API default

  Runs in:

  every verdict, plus the paid report's measured map; not the free measured map

• Claudeclaude-haiku-4-5

  API:

  Anthropic Messages API

  Temperature:

  0 (since 2026-08-23; API default before that)

  Max tokens:

  1200

  Runs in:

  every audit

• Perplexitysonar

  API:

  Perplexity chat completions API (web-grounded)

  Temperature:

  0

  Max tokens:

  API default

  Runs in:

  only when a funded API key is configured; not promised in marketing copy

The snapshot uses the same model ids with a system prompt that fixes the JSON format and the scale, JSON mode and temperature 0.2 on the OpenAI-compatible providers, and a 1024-token limit on Anthropic. We keep the two prompts separate on purpose: the snapshot asks the model to judge, the map asks the model the buyer's question. Only the second one is a measurement.

Why budget-tier models

These are the fast, cheap tiers of each family, not the flagship models. At list prices they cost between USD 0.40 and USD 1 per million input tokens and between USD 1.60 and USD 5 per million output tokens, and a free audit makes on the order of fifty short calls. That is what makes the audit free and, more importantly, repeatable: you can re-run it after you publish and compare. A flagship-only audit would cost enough that you would run it once, which is exactly the single measurement this page argues against. The trade-off is real and we state it: a flagship model may name different brands. If you need that comparison, the report shows which model answered, and the model ids can be changed per deployment.

We are not affiliated with any of these companies. Provider names refer to the model APIs we call, not the consumer chat products.

How many times

Snapshot: 3 runs per provider. The structured assessment is sent 3 times to each provider through a determinism harness that records every run, averages the result and keeps the run closest to the average as the representative answer. The count is set per deployment between 1 and 5; if our monthly AI budget is exceeded, the snapshot falls back to 1 run per provider so the free audit keeps working. Alongside the 3 assessment runs, each provider also answers one raw buyer question for your category, quoted on the report.

Buyer-question map: 1 run per question per provider. Up to 12 questions, read from your own site and written in its language, each sent once to each active provider. With 2 default providers that is up to 24 answers per audit. Some questions mention your brand, to see whether the model knows it; only the open questions, where the brand is not in the question, count toward the verdict, because being named when the buyer already named you is recognition, not a recommendation.

What each number on the report means

• "Recommended in X of N open buyer questions": N is the number of open questions that got an answer, X is how many of those answers named your brand without being prompted to. One answer per question per provider; an answer that failed or timed out is not in N.
• Per-provider counts: the same X of N, split by model, so you can see that one model names you and another does not.
• Competitor counts: how many answers named each competitor, one count per answer, not per mention. Names that are not brands (a category, a generic term) are filtered in code and then confirmed against your category in one extra model call.
• The verdict: Visible enough to optimize when recommended in at least 50% of the answered open questions; Underspecified in AI search when recommended in at least 15% but under 50%; Likely invisible when recommended in under 15%. It needs at least 6 answered open questions; below that the verdict falls back to the snapshot's estimate, which is the model judging itself and the weaker signal.

Why there is no score

We published a 0-100 visibility score until August 2026 and removed it from every screen on 2026-08-16. The reason is worth writing down: the models do not share a ruler. Asked about the same brand on the same day, one model answered 75, another 49 and another 38. Averaging three different rulers gives a number that looks precise and moves when nothing about the brand has moved.

It was worse before we noticed that the prompt never stated a scale at all: one provider was answering with a probability between 0 and 1, which rounded to 1 out of 100 while the same response said the brand does get named. Fixing that cut the swing between identical runs by two thirds. It still left a number we would not defend to a buyer, so it is gone.

The number is still computed and stored, because the dataset and the public index need a comparable quantity. Scoring is versioned (currently version 2): when the scoring prompt changes, the version goes up, and the index only compares observations measured under one version. What you see instead can be counted and re-counted: the three-band verdict, how many of the open questions each model answers with you, a competitor, or nobody, and the raw answers behind each count.

What a count can and cannot tell you

A count of 3 of 12 says that on the day we asked, three of twelve answers named you. It does not say you will be named tomorrow, and it does not say that 3 is different from 4: with one run per question, one answer flipping moves the count by one. The snapshot's repeated runs show how much the same model moves on the same input; with 3 runs, appearance can only read as 0, 33, 67 or 100 percent, and the confidence interval we compute on it is wide. That is expected, and it is what the interval is there to say.

What a count can tell you: who is being named instead of you, in which questions, by which model, with the answer text to check. And, across re-runs, whether a change you published moved anything by more than the run-to-run noise. That is why the first run of every audit is free, including a re-run of a brand you measured before. The full report costs USD 9 once per audit; the lifetime pass is USD 29.99 once and unlocks the report on every audit and re-run after that. No subscription.

Independent research we align with

We did not invent the problem. These are the studies whose findings match what we see in our own re-runs, and they are the reason the method looks the way it does.

• AI recommendation lists rarely repeat

  SparkToro and Gumshoe, as reported by Search Engine Land

  about 3,000 runs; identical recommendation lists under 1 percent of the time. Same conclusion we reached with our own re-runs: one answer is a sample, not the answer.

• Don't Measure Once: Measuring Visibility in AI Search (GEO)

  Schulte, Bleeker, Kaufmann (arXiv 2604.07585, April 2026)

  its title is the rule we follow: the snapshot repeats its prompt, and we tell you to re-run rather than trust one map.

• Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement

  Ronald Sielinski (arXiv 2603.08924, March 2026, revised June 2026)

  frames AI visibility as an estimate with uncertainty, as the title says. We report counts and refuse a single score for the same reason.

• How accurate are AI visibility tools

  Petra Labs

  visibility differences of up to 32 points between logged-in ChatGPT and the API. We measure the API and say so; we do not claim to read the consumer app.

Our own study: How repeatable are AI recommendations? 4,500 answers from three model APIs. The same buyer question asked 5 times to the same model returned the same set of brands in 15.0 percent of cases (mean Jaccard 0.54); the first-named brand held in 75 percent of runs; half of all brands named (49.8 percent) appeared only once. Dataset and raw answers published under CC BY 4.0.

How the public AI Index is built

The AI Index aggregates audits into a per-category ranking. The order comes from the average of a brand's observations, but the underlying number is not published, for the same reason it is not shown in your own report: it is not stable enough to print next to somebody else's name. Brands are shown in three tiers; brands closer together than a single brand moves between its own runs share a tier. A category is published only once it has at least 5 brands and 10 observations. The number of measurements next to each brand is the raw count of observations behind it, so you can see how much a position rests on.

Only observations backed by all 3 providers enter the index. Scores rise with the number of providers that answered, so an audit where one provider timed out or hit a rate limit scores systematically lower for reasons that have nothing to do with the brand. Mixing those into a ranking would measure our own collection reliability and present it as brand visibility. Partial audits are still shown in your own report, where the per-provider breakdown makes the gap visible; they are excluded here, where brands are compared against each other.

A brand needs at least 2 complete observations to be ranked. AI answers are not deterministic: three identical runs of the same brand have returned 45, 30 and 29 here. A single run is a coin flip, and until this rule existed a brand could hold a top position on one. Two runs is a low bar and we say so plainly: it does not make an average precise, it only means one run is not enough to rank someone else's company.

One scoring version per edition. Observations carry the scoring version they were measured under (currently 2); an edition is built from one version only, so a change in the scoring prompt never mixes two rulers in one ranking. The consequence of all these rules is that the index is smaller than our dataset, and categories and brands drop out of it when their complete measurements fall below the thresholds. We prefer a narrower index we can defend to a broader one we cannot.

Asked Thrice does not rank itself in this index. Not for modesty: our sample is not collected the same way. We re-run our own audit every time we test a change, so we accumulate far more observations than a brand we measured two or three times, under conditions we chose. An average over that sample is not comparable to a competitor's, and it would come from the one company that also controls the method and decides when to measure. We would rather rank a market we are not in than defend a number nobody can check.

Changelog of this page

• 2026-08-25The free measured map runs OpenAI and Claude; Gemini still answers every verdict, and still runs the paid report's measured map. The map is where the cost is: one free map costs ten times one verdict, so this keeps the free tier sustainable without dropping a model from the verdict. Counts always name the models that actually answered.
• 2026-08-23Claude map calls now run at temperature 0 like every other provider. Our repeatability study (4,500 calls, 30 categories) found Claude at the API default temperature repeated the same set of brands in 5 percent of cells against 35 percent for OpenAI at temperature 0; that gap was the parameter, not the model. Counts measured before this date are not directly comparable for Claude.
• 2026-08-23Page rewritten: exact models, temperatures and token limits, the snapshot-versus-map run counts, the research we align with, and this changelog.
• 2026-08-16The 0-100 score left every screen. The verdict is three bands plus counts from the measured map; the number is still computed and stored for the dataset and the index.
• 2026-08-15Live buyer-question map introduced: each question from your own site is asked to every active provider and the answers are stored as evidence.
• 2026-08-14AI Index republished with two new rules: observations need every default provider, and a brand needs two complete observations to be ranked.
• 2026-07-26First public edition of the AI Index.