Shopify AI Crawler Access: Complete Technical Guide

Crawler access is not one switch. A Shopify URL passes through DNS, TLS, a CDN or WAF, Shopify routing, redirects, robots rules, theme rendering, and sometimes client-side code. A failure at any layer can make an otherwise excellent product page unusable as evidence.

Test the response before the policy

Start with the exact canonical product URL:

curl -sSIL --max-redirs 10 https://example.com/products/example-product
curl -sSL -o /tmp/product.html -w '%{http_code} %{url_effective}\\n' \
  https://example.com/products/example-product

Record:

• final status
• redirect chain
• effective URL
• content type
• response size
• canonical link in the HTML

A robots rule is irrelevant if the crawler first receives a 403, challenge page, redirect loop, or empty response.

Inspect Shopify robots rules

curl -sS https://example.com/robots.txt

Review every matching user-agent group. A specific group takes precedence differently than a broad User-agent: * group, so do not audit by searching only for Disallow.

Shopify generates a default robots.txt suitable for most stores. If customization is necessary, create templates/robots.txt.liquid and preserve Shopify’s default groups rather than replacing the entire file with a static example.

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {% for rule in group.rules %}
    {{- rule }}
  {% endfor %}
  {%- if group.sitemap != blank %}
    {{ group.sitemap }}
  {% endif %}
{% endfor %}

Add explicit rules only after verifying the crawler’s official documentation. User-agent names and operational behavior can change.

Distinguish indexing, search retrieval, and user fetches

AI products may operate more than one crawler:

• a crawler that builds or refreshes a search index
• a user-triggered fetcher that opens a URL during a session
• a separate bot used for model training or other purposes

One allow rule does not imply participation in every product. Conversely, blocking a training crawler does not necessarily block search retrieval. Follow the provider’s current documentation and verify WAF traffic using both documented user agents and, where provided, published IP ranges.

Perplexity, for example, publishes crawler descriptions and recommends validating IP ranges in addition to user-agent matching. Treat a user-agent string alone as spoofable.

Check Shopify’s agent-discovery routes

As of May 2026, Shopify documents these storefront routes:

• /agents.md
• /llms.txt
• /llms-full.txt

By default, the latter routes can mirror the agents.md content. A theme can provide separate Liquid templates.

for path in agents.md llms.txt llms-full.txt; do
  curl -sSIL "https://example.com/$path" | sed -n '1,8p'
done

These files are supplementary. They do not override robots.txt, repair a blocked CDN, or make a product page accurate. Keep them concise and link to canonical collections, policies, and products.

Test the infrastructure layer

Common CDN and WAF failures include:

• bot-fight rules that challenge all non-browser traffic
• rate limits shared across unrelated crawler requests
• country blocks that redirect to unsupported markets
• TLS settings that work in one browser but fail for other clients
• cache rules serving stale stock or redirects

Use server or Cloudflare logs when available. Record response status by path, user agent, country, ASN, and rule ID. Never whitelist a claimed crawler solely because its user-agent string looks legitimate.

Confirm that facts exist in returned HTML

A 200 response can still be unusable if the initial HTML lacks product facts and scripts fail to render.

curl -sSL https://example.com/products/example-product |
  rg -n '<title|rel=\"canonical\"|application/ld\\+json|price|availability|sku'

Check:

• meaningful title and description
• product name and decisive specifications
• canonical URL
• product structured data
• links to policies and related content

Client-side enhancements are acceptable; critical product identity should not depend entirely on an interaction.

Use an access test matrix

Path	Normal fetch	Documented crawler	Expected
/	200	200	accessible
priority product	200	200	accessible
collection	200	200	accessible
/robots.txt	200	200	plain text
/sitemap.xml	200	200	XML or sitemap index
/agents.md	200	200	concise Markdown
cart/account	varies	restricted	not an evidence target

Run from more than one network if regional controls are active.

Safe remediation order

1. Fix DNS, TLS, and status failures.
2. Remove unintended WAF challenges.
3. Correct redirects and canonicals.
4. Fix robots rules.
5. Ensure critical facts exist in HTML.
6. Add or refine agent-discovery files.
7. retest and retain response evidence.

Do not begin by adding llms.txt while product pages return challenges. Access work should follow the request path.

Primary references

• Shopify robots.txt.liquid
• Shopify agent-discovery templates
• Perplexity crawlers
• Google AI features and technical requirements