AI crawler and user-agent reference for Shopify stores

Crawler names and policies change. Review the linked first-party documentation before changing robots.txt.liquid or a WAF rule.

Current role separation

Provider	User agent	Documented role	Do not infer
OpenAI	OAI-SearchBot	search discovery	guaranteed ChatGPT citation
OpenAI	ChatGPT-User	user-triggered page access	model-training policy
OpenAI	GPTBot	separately documented model-improvement use	search visibility
Perplexity	PerplexityBot	search result discovery	guaranteed recommendation
Google	Googlebot	Search crawling and indexing	control by Google-Extended
Google	Google-Extended	separate generative-AI control token	ordinary Search indexing
Anthropic	Claude-SearchBot	search-result quality and relevance	every Claude answer uses the page
Anthropic	Claude-User	user-directed retrieval	training access
Anthropic	ClaudeBot	model-development crawling	Claude search visibility

Primary documentation:

• OpenAI crawler documentation
• Perplexity crawler documentation
• Google AI features and controls
• Anthropic crawler documentation

Synthetic User-Agent reachability test

On July 27, 2026, SkuWatch AI Visibility sent its own HTTP requests to selected public product URLs from The Woobles, Rhode, Ring Australia, HexClad UK, and MyFonts. The requests substituted crawler-associated User-Agent header values. They did not originate from provider IP space and were not authenticated as provider crawlers.

Property	What was recorded
Request origin	SkuWatch AI Visibility research infrastructure
Request type	Synthetic HTTP request with a substituted User-Agent header
Transport result	Final HTTP status was 200 for the recorded URL/header combinations
Identity established	None; User-Agent text is spoofable
Provider crawl established	No
Indexing, citation, or recommendation established	No

Status 200 is not sufficient to call the body usable. A repeatable test must also store the final URL, redirect chain, content type, response size or hash, expected product-title presence, canonical, blocking-page indicators, and relevant robots directives. Without those body checks, the correct conclusion is only that the server returned a nominally successful response to that synthetic request.

Actual provider access requires separate evidence such as verified infrastructure logs. Actual visibility requires a completed provider answer with the exact prompt, model, time, answer text, and citation metadata.

Robots evaluation

Retrieve the public file:

curl -sS https://example.com/robots.txt

For each crawler:

1. locate the most specific matching group
2. apply allow/disallow precedence correctly
3. test the exact public product path
4. inspect CDN/WAF behavior separately
5. verify legitimate traffic using provider-supported methods

Ring Australia’s observed robots file grouped several AI agents, allowed /, and disallowed account, admin, checkout, cart, and sign-in paths. This is a useful public-versus-private policy pattern, not universal copy-paste text.

User-agent comparison

url='https://example.com/products/example-product'

for agent in \
  'OAI-SearchBot/1.0; +https://openai.com/searchbot' \
  'ChatGPT-User/1.0' \
  'PerplexityBot/1.0; +https://perplexity.ai/perplexitybot' \
  'Claude-SearchBot' \
  'Claude-User' \
  'Googlebot/2.1'; do
  curl -sSL -A "$agent" -o /dev/null \
    -w "$agent -> %{http_code} %{url_effective} %{size_download}\n" \
    "$url"
done

Verification boundaries

• a user-agent header can be spoofed
• a 200 can contain a challenge or password page
• access does not prove indexing
• indexing does not prove citation
• citation does not prove recommendation position
• training controls and search controls are not interchangeable

Perplexity publishes IP verification guidance. Anthropic currently states that it does not publish stable IP ranges and documents robots controls. Apply each provider’s method rather than inventing one global allowlist.

Shopify policy checklist

• public products, collections, policies, and editorial evidence intentionally reachable
• cart, checkout, admin, and customer data remain protected
• custom robots.txt.liquid preserves Shopify defaults
• WAF rules do not trust a header alone
• changes have owner, date, rationale, and rollback
• sampled answer observations remain separate from crawler checks