Why Provider Failures Must Be Excluded, Not Scored as Zero

Suppose a visibility monitor schedules 100 prompts. Twenty requests time out, ten are rate-limited, and the remaining 70 return answers. The brand appears in 14 completed answers.

Calling that a 14% visibility rate mixes two different systems:

• the provider or scanner’s ability to complete a run
• the brand’s presence in a completed answer

The defensible observation is:

• completion rate: 70%
• mention rate among completed answers: 20%
• 30 runs produced no visibility evidence

The state model

Every run receives one terminal state:

success_mention
success_no_mention
refused
rate_limited
timeout
provider_error
parser_error
cancelled

Only the first two belong in the mention-rate denominator.

Why zero is harmful

Scoring infrastructure failures as absence can:

• make a provider outage look like a storefront regression
• reward systems with fewer attempted runs
• hide a broken parser behind a “visibility decline”
• cause merchants to change pages that were never evaluated
• make before-and-after tests incomparable

Missing evidence is not positive evidence of absence.

Store raw observations

A run record needs:

{
  "query_id": "doorbell-au-compatibility-01",
  "provider": "example-provider",
  "scheduled_at": "2026-07-27T15:00:00Z",
  "state": "rate_limited",
  "http_status": 429,
  "attempts": 1,
  "answer": null,
  "mentions": null,
  "citations": null
}

Use null, not empty arrays, when an answer was never available to inspect. An empty mention array means a successful answer contained no detected mention.

Report both denominators

Scheduled: 100
Completed: 70
Mentioned: 14

Completion rate = 70 / 100 = 70%
Observed mention rate = 14 / 70 = 20%

If completion is low, mark the visibility estimate as low confidence. Do not silently publish it as comparable to a period with 99% completion.

Retry without bias

Retry policy must be identical for mention and no-mention candidates. Because mention is unknown until an answer succeeds, retry based on technical state only:

• exponential backoff for rate limits
• bounded retry count
• provider-specific timeout
• no prompt mutation during a retry
• retain all attempts

Parser failures are first-class

If an answer exists but citation extraction fails, preserve the answer and mark citation scoring unavailable. Do not classify it as “no citation.”

The same principle applies to screenshots, regional page fetches, and structured-data parsing.

Dashboard rules

Every visibility chart should show:

• completed-run denominator
• completion rate
• provider-error distribution
• query coverage
• confidence or minimum observation-count warning

This design is less dramatic than one universal score. It is also much harder to misinterpret, which is the point.