Building Repeatable Buyer-Prompt Monitoring

A prompt is configuration, not free-form text entered before every run.

{"query_id":"doorbell-au-01","version":3,"locale":"en-AU","prompt":"Battery doorbells for 2.4 GHz Wi-Fi without existing wiring","active_from":"2026-07-01"}

Run pipeline

schedule → provider adapter → terminal state → raw observation → source extraction → entity matching → fact checks → aggregate.

Retries preserve query version and do not mutate wording. Rate limits and timeouts remain separate. Expected commercial facts are refreshed before a run; the old expected snapshot remains attached to the old observation.

Drift

Changing prompt wording creates a new version and comparison boundary. Provider UI/model changes are metadata, not silently normalized away.

Output

Show completed denominator, official citations, brand-only mentions, fact accuracy, provider failures, and named competitors. Never collapse these into one score.

Prompt portfolio design

A useful portfolio maps buyer intent to products and constraints. It is not a list of brand-name vanity queries. For a snowboard catalog, separate all-mountain, powder, beginner, size, terrain, budget, and compatibility questions. Each query records why it exists and which facts are required to judge an answer.

{
  "query_id": "snowboard-all-mountain-ca-01",
  "version": 2,
  "intent": "comparison",
  "market": "CA",
  "language": "en",
  "constraints": ["all-mountain", "intermediate", "CAD 700 maximum"],
  "expected_fact_snapshot_id": "facts_01J...",
  "prompt": "Which all-mountain snowboard under CAD 700 suits an intermediate rider who spends some days in powder?"
}

Brand names stay out of discovery prompts unless the test is explicitly about brand recall or comparison. Otherwise the measurement is biased toward producing the desired mention.

Terminal provider states

Each provider adapter must end in one of a small number of explicit states: completed, rate_limited, timed_out, tool_unavailable, blocked, invalid_response, or configuration_error. Only completed responses enter an appearance-rate denominator. A timeout is not a non-mention.

Store the provider, model or endpoint identifier, tool mode, locale context, request time, response time, answer text, visible citations, extracted search queries when supplied, and response identifier. Do not normalize different consumer interfaces and APIs into the same surface label.

Comparison boundaries

Comparisons are valid only when the query version, market, language, evidence rules, and completion policy are compatible. If wording changes, start a new series. If a provider changes its search tool or citation format, annotate the boundary. Historical answers remain immutable.

A release comparison can report:

before: 8 completed / 2 official storefront citations / 1 brand-only mention
after:  9 completed / 3 official storefront citations / 1 brand-only mention
failed: 1 before / 0 after

It cannot claim that the storefront change caused the difference without stronger evidence.

Verification fixtures

Test adapters with a completed answer, no citations, duplicated citations, provider failure, redirected source, malformed payload, and answer mentioning an ambiguous brand. Confirm that retries retain the same query version and produce a new attempt record rather than overwriting the first attempt.

The static website publishes observations only after a manual collection and editorial review. It does not run a website-content cron job or automatically publish provider output. See how to measure before and after for the merchant workflow.