# How Asked Thrice works

> Every number on a report comes from a call we can name: which model, which parameters, how many times. This page lists them, explains what the counts can and cannot tell you, and states what we do not claim. Last substantive change: 2026-08-23.

Canonical page: https://askedthrice.com/methodology

## At a glance

- What it is: a measured count of AI answers. Not a Google rank and not a score.
- Models queried: gpt-4.1-mini, claude-haiku-4-5 (OpenAI and Claude), through their model APIs, with no web search.
- How many times: snapshot, 3 runs per provider; buyer-question map, 1 run per question per provider.
- What we report: a three-band verdict, X-of-N counts, the raw answers, the competitor gap. There is no 0-100 score.

## What we measure

We measure the model APIs: the same OpenAI, Google and Anthropic models that power the assistants, called directly, with the buyer's question as the only input. We do not log in to ChatGPT, Gemini or Claude as a consumer and read what the app shows, and we do not scrape those apps.

The distinction matters. The consumer apps add their own system prompts, memory, personalization and, when browsing is on, a web search step before the model answers. Petra Labs measured differences of up to 32 points in brand visibility between logged-in ChatGPT and the API for the same questions. A count on this site is a count of what the model says on its own, with no retrieval layer in front of it, and that is what we claim, no more.

Two things are measured in every free audit. The snapshot is a structured assessment: each model is given the brand, category and prompts and asked to judge, in JSON, whether it would name the brand and what is missing. The buyer-question map is the measurement: the questions a buyer in the category would ask, read from the brand's own site, asked raw to every active provider, with the answers stored as evidence and the brands named counted in code. The verdict and every count on the report come from the map.

## Which models, exactly

Every live question is sent as a single user message, with no system prompt and no tools. No web search, browsing or grounding is enabled on any provider except Perplexity, whose product is web-grounded by design. Each map call has a 20-second timeout. Model ids are the defaults in our code; an environment variable can pin a newer id, and the report names the model that actually answered.

- OpenAI: gpt-4.1-mini. API: OpenAI Chat Completions API. Temperature: 0. Max tokens: API default. Runs in: every audit.
- DeepSeek: deepseek-v4-flash. API: DeepSeek chat completions API. Temperature: 0. Max tokens: API default. Runs in: off by default; opt-in flag, not part of any published measurement.
- Google Gemini: gemini-flash-latest. API: Gemini API, OpenAI-compatible chat completions endpoint. Temperature: 0. Max tokens: API default. Runs in: every verdict, plus the paid report's measured map; not the free measured map.
- Claude: claude-haiku-4-5. API: Anthropic Messages API. Temperature: 0 (since 2026-08-23; API default before that). Max tokens: 1200. Runs in: every audit.
- Perplexity: sonar. API: Perplexity chat completions API (web-grounded). Temperature: 0. Max tokens: API default. Runs in: only when a funded API key is configured; not promised in marketing copy.

The snapshot uses the same model ids with a system prompt that fixes the JSON format and the scale, JSON mode and temperature 0.2 on the OpenAI-compatible providers, and a 1024-token limit on Anthropic. The two prompts are kept separate on purpose: the snapshot asks the model to judge, the map asks the model the buyer's question. Only the second one is a measurement.

### Why budget-tier models

These are the fast, cheap tiers of each family, not the flagship models. At list prices they cost between USD 0.40 and USD 1 per million input tokens and between USD 1.60 and USD 5 per million output tokens, and a free audit makes on the order of fifty short calls. That is what makes the audit free and, more importantly, repeatable: you can re-run it after you publish and compare. A flagship-only audit would cost enough that you would run it once, which is exactly the single measurement this page argues against. The trade-off is real: a flagship model may name different brands. The report shows which model answered, and the model ids can be changed per deployment.

We are not affiliated with any of these companies. Provider names refer to the model APIs we call, not the consumer chat products.

## How many times

Snapshot: 3 runs per provider. The structured assessment is sent 3 times to each provider through a determinism harness that records every run, averages the result and keeps the run closest to the average as the representative answer. The count is set per deployment between 1 and 5; if our monthly AI budget is exceeded, the snapshot falls back to 1 run per provider so the free audit keeps working. Alongside the 3 assessment runs, each provider also answers one raw buyer question for the category, quoted on the report.

Buyer-question map: 1 run per question per provider. Up to 12 questions, read from the brand's own site and written in its language, each sent once to each active provider. With 2 default providers that is up to 24 answers per audit. Some questions mention the brand, to see whether the model knows it; only the open questions, where the brand is not in the question, count toward the verdict, because being named when the buyer already named you is recognition, not a recommendation.

### What each number on the report means

- "Recommended in X of N open buyer questions": N is the number of open questions that got an answer, X is how many of those answers named the brand without being prompted to. One answer per question per provider; an answer that failed or timed out is not in N.
- Per-provider counts: the same X of N, split by model.
- Competitor counts: how many answers named each competitor, one count per answer, not per mention. Names that are not brands are filtered in code and then confirmed against the category in one extra model call.
- The verdict, from the share of answered open questions that recommended the brand:
  - Visible enough to optimize: recommended in at least 50% of the answered open questions.
  - Underspecified in AI search: recommended in at least 15% but under 50%.
  - Likely invisible: recommended in under 15%.
  It needs at least 6 answered open questions; below that the verdict falls back to the snapshot's estimate, which is the model judging itself and the weaker signal.

## Why there is no score

We published a 0-100 visibility score until August 2026 and removed it from every screen on 2026-08-16, because the models do not share a ruler. Asked about the same brand on the same day, one model answered 75, another 49 and another 38. Averaging three different rulers gives a number that looks precise and moves when nothing about the brand has moved.

It was worse before we noticed that the prompt never stated a scale at all: one provider was answering with a probability between 0 and 1, which rounded to 1 out of 100 while the same response said the brand does get named. Fixing that cut the swing between identical runs by two thirds. It still left a number we would not defend to a buyer, so it is gone.

The number is still computed and stored, because the dataset and the public index need a comparable quantity. Scoring is versioned (currently version 2): when the scoring prompt changes, the version goes up, and the index only compares observations measured under one version. What you see instead can be counted and re-counted: the three-band verdict, how many of the open questions each model answers with you, a competitor, or nobody, and the raw answers behind each count.

## What a count can and cannot tell you

A count of 3 of 12 says that on the day we asked, three of twelve answers named you. It does not say you will be named tomorrow, and it does not say that 3 is different from 4: with one run per question, one answer flipping moves the count by one. The snapshot's repeated runs show how much the same model moves on the same input; with 3 runs, appearance can only read as a few coarse fractions, and the confidence interval we compute on it is wide. That is expected, and it is what the interval is there to say.

What a count can tell you: who is being named instead of you, in which questions, by which model, with the answer text to check. And, across re-runs, whether a change you published moved anything by more than the run-to-run noise. That is why the first run of every audit is free, including a re-run of a brand you measured before. The full report costs USD 9 once per audit; the lifetime pass is USD 29.99 once and unlocks the report on every audit and re-run after that. No subscription.

## Independent research we align with

We did not invent the problem. These are the studies whose findings match what we see in our own re-runs, and they are the reason the method looks the way it does.

- AI recommendation lists rarely repeat (SparkToro and Gumshoe, as reported by Search Engine Land): about 3,000 runs; identical recommendation lists under 1 percent of the time. Same conclusion we reached with our own re-runs: one answer is a sample, not the answer. https://searchengineland.com/ai-recommendation-lists-rarely-repeat-study-468076
- Don't Measure Once: Measuring Visibility in AI Search (GEO) (Schulte, Bleeker, Kaufmann (arXiv 2604.07585, April 2026)): its title is the rule we follow: the snapshot repeats its prompt, and we tell you to re-run rather than trust one map. https://arxiv.org/abs/2604.07585
- Quantifying Uncertainty in AI Visibility: A Statistical Framework for Generative Search Measurement (Ronald Sielinski (arXiv 2603.08924, March 2026, revised June 2026)): frames AI visibility as an estimate with uncertainty, as the title says. We report counts and refuse a single score for the same reason. https://arxiv.org/abs/2603.08924
- How accurate are AI visibility tools (Petra Labs): visibility differences of up to 32 points between logged-in ChatGPT and the API. We measure the API and say so; we do not claim to read the consumer app. https://www.petralabs.com/intelligence/how-accurate-are-ai-visibility-tools

Our own study: How repeatable are AI recommendations? 4,500 answers from three model APIs. The same buyer question asked 5 times to the same model returned the same set of brands in 15.0 percent of cases (mean Jaccard 0.54); the first-named brand held in 75 percent of runs; half of all brands named (49.8 percent) appeared only once. Dataset and raw answers published under CC BY 4.0: https://askedthrice.com/research/how-repeatable-are-ai-recommendations.md

## How the public AI Index is built

The AI Index (https://askedthrice.com/ai-index) aggregates audits into a per-category ranking. The order comes from the average of a brand's observations, but the underlying number is not published, for the same reason it is not shown in your own report. Brands are shown in three tiers; brands closer together than a single brand moves between its own runs share a tier.

- A category is published once it has at least 5 brands and 10 observations. The number of measurements next to each brand is the raw count of observations behind it.
- Only observations backed by all 3 providers enter the index. A partial audit scores lower for reasons that have nothing to do with the brand.
- A brand needs at least 2 complete observations to be ranked. Three identical runs of the same brand have returned 45, 30 and 29 here; one run is not enough to rank someone else's company.
- One scoring version per edition (currently 2), so a change in the scoring prompt never mixes two rulers in one ranking.
- Asked Thrice does not rank itself in this index: our own sample is collected under conditions we chose, so it is not comparable.

## What we do not claim

- We do not claim to query models we do not actually call. The list above is the list.
- We do not read the consumer ChatGPT, Gemini, Claude or Perplexity apps. We call the model APIs, and answers inside an app with browsing, memory or personalization can differ.
- We do not run web search, browsing or grounding on OpenAI, Gemini or Anthropic calls. A count here is what the model says on its own.
- We do not present a single run as ground truth. One answer is a sample; confidence comes from repetition and re-runs.
- We do not guarantee rankings, traffic, citations or future model behavior.
- We do not publish a 0-100 score, and we do not rank ourselves in our own index.
- We are not affiliated with OpenAI, Google, Anthropic, Perplexity or DeepSeek.

## Changelog of this page

- 2026-08-25: The free measured map runs OpenAI and Claude; Gemini still answers every verdict, and still runs the paid report's measured map. The map is where the cost is: one free map costs ten times one verdict, so this keeps the free tier sustainable without dropping a model from the verdict. Counts always name the models that actually answered.
- 2026-08-23: Claude map calls now run at temperature 0 like every other provider. Our repeatability study (4,500 calls, 30 categories) found Claude at the API default temperature repeated the same set of brands in 5 percent of cells against 35 percent for OpenAI at temperature 0; that gap was the parameter, not the model. Counts measured before this date are not directly comparable for Claude.
- 2026-08-23: Page rewritten: exact models, temperatures and token limits, the snapshot-versus-map run counts, the research we align with, and this changelog.
- 2026-08-16: The 0-100 score left every screen. The verdict is three bands plus counts from the measured map; the number is still computed and stored for the dataset and the index.
- 2026-08-15: Live buyer-question map introduced: each question from your own site is asked to every active provider and the answers are stored as evidence.
- 2026-08-14: AI Index republished with two new rules: observations need every default provider, and a brand needs two complete observations to be ranked.
- 2026-07-26: First public edition of the AI Index.

## Read next

- Run the free audit: https://askedthrice.com/
- What the report costs: https://askedthrice.com/pricing.md
- The public index: https://askedthrice.com/ai-index

---

Machine-readable index: https://askedthrice.com/llms.txt · Full context: https://askedthrice.com/llms-full.txt · All URLs: https://askedthrice.com/sitemap.xml · Developer and agent resources: https://askedthrice.com/developers.md