Methodology version 2.1 · 13 August 2026

A frozen evidence panel—not a claim about what “AI thinks.”

This is the proposed protocol for the limited commercial pilot. It has not yet been run end to end. The historical example uses a separate legacy design and is never blended into a current pilot panel.

1. Measurement target

The primary finite-panel estimand is: for the versioned set of non-branded buying questions, named answer surfaces, locale, and collection window agreed with the buyer, how often does a reviewed brand or product entity appear in the natural recommendation set?

Finite-panel rate: a descriptive result for the exact frozen prompts, surfaces, locale, and windows. Superpopulation rate: an estimate beyond that panel requires a documented sampling frame and probability-based prompt/window sampling. Without that design, resampling output is a robustness interval—not a population confidence interval.

Consumer chat products, search-grounded interfaces, APIs, locales, and memory-only models are separate panels unless an explicit weighting rule is frozen in advance.

2. Core reported measures

Measure	Definition and boundary
Panel shortlist presence	Eligible answers containing the reviewed entity divided by all eligible answers in the declared panel. Never described as market share.
First-listed rate	Answers naturally placing the entity first divided by all eligible answers. Reported only when ordering is clear.
Unconditional MRR	Mean reciprocal rank with zero when absent, so absence is not hidden by an appearance-only mean rank.
Top-k presence	A standardized cut alongside any-list presence, because answer-set length differs across surfaces.
Answer-set size and refusal	Recommendation count plus refusal/no-recommendation rate, reported by surface.
Prompt and surface sensitivity	Range and dispersion across predeclared intent strata and surfaces; not smoothed into a proprietary score.
Selected claim status	Confirmed, contradicted, stale, unsupported, unresolved, or not reviewed against dated sources with jurisdiction and applicability recorded.

3. Proposed diagnostic design

20 buyer-approved non-branded prompts across discovery, use case, hard constraint, comparison, and switching intents.
Four explicitly named answer surfaces and one buyer-relevant locale.
Five randomized fresh sessions per prompt × surface cell inside a seven-day window: 400 nominal answers.
Natural answer requests: no forced five-product schema; allow zero, one, or many recommendations.
Every answer begins in an isolated session. Record product/surface, model label where exposed, search/tool state, locale, timestamp, collector/parser version, request status, retry, and source-answer identifier.
Randomize the execution queue and balance every declared cell to the same repeat target. A transport failure is missing—not a non-mention.

The free fit check performs no collection. It qualifies one SKU, category, market, business decision, and evidence risk before a paid scope is offered.

4. Uncertainty and change

Repeated answers from the same prompt, surface, and time are correlated. The pilot reports raw numerators/denominators and a stratified cluster-resampling robustness analysis over prompts, windows, and replicate sessions. It does not label that output a population confidence interval unless a defensible sampling frame exists.

Estimate intraclass correlation and effective sample size; repeated sessions reveal stochasticity but do not replace independent prompts or windows.
No per-prompt probability claim from three to five repeats.
Predeclare primary endpoints and the number of comparisons. Exploratory multi-entity/surface cuts are labelled exploratory; alerts require false-discovery control or an independent confirmation panel.
No “category leader” label until a false-leader simulation/backtest calibrates the rule. Otherwise report a tied or unresolved leading set.
No causal claim from a before/after chart. Causal tests need a preregistered intervention and matched or randomized comparison.

5. Product and claim review

Entity mapping preserves what the answer wrote while aggregating only under a versioned, dated registry. Exact matches are preferred. Fuzzy matching may propose a candidate but cannot silently merge it.

Track company, brand, family, exact SKU, variant, predecessor, successor, discontinued, recalled, and unresolved states separately.
Maintain a reviewed gold set and report entity precision, recall, unresolved rate, ambiguous-family rate, and sensitivity of headline metrics to contested mappings.
A model-family name never implies a serial or lot number.
Material-claim review uses a written codebook, source date, jurisdiction/applicability, excerpt, reviewer, and outcome. Severe findings require dual review and adjudication.
“Verified” is reserved for completed, recorded review; otherwise use “checked,” “unresolved,” or “not reviewed.”

6. Quality gates

At least 95% cell completion overall and 90% in every surface block, or no blended result.
Immutable raw evidence; corrections append a new interpretation version rather than rewriting the source answer.
Schema validation and explicit extraction/repair tier for every parsed record.
Manual review of every material public claim and every entity merge affecting a reported top set.
Publish exclusions, unresolved mappings, missingness, methodology version, report version, reviewer, and last-checked date.

7. What the legacy audit found

Reproduced local fact	Why it blocks reuse as current intelligence
6,056 stored runs; 6,028 parsed; two tainted; one failed; 6,026 passed the legacy flags.	Passing parser/tool/failure flags does not establish factual accuracy or representative sampling.
One Claude Haiku 4.5 CLI configuration across five dates.	Not a current cross-surface buyer-experience panel.
All 6,028 parseable answers contained exactly five picks.	The forced schema changes answer length and rank behavior.
68 of 345 complete July 2 category × wording cells retained the same rank-one string across three repeats.	A screenshot or three calls cannot support a stable-winner claim.
42 of 108 displayed guides had tied top mention counts; 92 had a lead of no more than two calls.	The legacy public presentation overstated winner certainty.
Ten clean runs contained duplicate normalized entities.	String normalization is not a reviewed product graph.

Database integrity check passed. Inspected snapshot SHA-256: 9dcc9357eabc4c62087cb064a68121969c954634df96477dc24fcbc1bac103fb. The historical BowFlex case was selected after corpus collection and is labelled retrospective.

8. Scope, conflicts, and corrections

Initial pilot excludes health, child-safety, finance, cybersecurity, and other consequential categories.
Payment cannot change source answers, exclusions, match decisions, claim status, or reported results. Buyers may approve panel scope before collection; all scope choices are disclosed.
No placement, traffic, conversion, revenue, or optimization-outcome guarantee.
BotPicked is not affiliated with named brands, model providers, retailers, or regulators unless expressly stated.
Correction requests are reviewed against retained evidence. Material corrections receive a dated version note.

Request a correction at hello@botpicked.com. See also privacy and pilot terms.