How to Test AI Crawler Access to a Shopify Store with curl

What this check tells you

These requests show how the public storefront responds to named user agents at a specific time. They do not prove that a platform indexed the page or will use it in an answer.

Step 1: check the normal public response

Replace the example URL:

PRODUCT_URL='https://example.com/products/example-product'
curl -sSIL --max-time 20 "$PRODUCT_URL"

Record every status and the final destination. A successful final response normally returns 200 and content-type: text/html.

Step 2: repeat with relevant user agents

curl -sSIL --max-time 20 -A 'GPTBot' "$PRODUCT_URL"
curl -sSIL --max-time 20 -A 'ChatGPT-User' "$PRODUCT_URL"
curl -sSIL --max-time 20 -A 'PerplexityBot' "$PRODUCT_URL"
curl -sSIL --max-time 20 -A 'Mozilla/5.0' "$PRODUCT_URL"

Compare the final status, redirect path, content type, and response size. A bot-specific 403, challenge page, or unexpected redirect is worth investigating. Do not assume that different content means malicious cloaking; consent, localization, security, and CDN configuration can also change a response.

Step 3: inspect robots.txt

curl -sS -D robots-headers.txt https://example.com/robots.txt -o robots.txt
cat robots-headers.txt
cat robots.txt

Read the most specific user-agent group that applies. Check both explicit disallow rules and broader path rules. A missing explicit allow is not automatically a block; evaluate the complete robots policy.

Step 4: test the referenced paths

If robots.txt names a sitemap, request it. Also request agent-facing routes used by the store:

curl -sSIL https://example.com/sitemap.xml
curl -sSIL https://example.com/agents.md
curl -sSIL https://example.com/llms.txt

Then fetch the body and inspect the product, collection, policy, shipping, returns, and contact links. A readable guide that points to missing pages is not a successful implementation.

Expected result

• Stable final public URL.
• No password page or security challenge for intended public content.
• HTML returned for the product page.
• Robots policy consistent with the merchant’s intended crawler access.
• Sitemap and agent-facing files link to reachable canonical pages.

Common failures

• Storefront password protection.
• Redirect loop between locale or primary-domain settings.
• CDN bot challenge returns HTML with status 200.
• robots.txt blocks a shared product path.
• Sitemap redirects to an unexpected hostname.
• Agent file contains Theme markup or stale URLs.
• Only the home page is reachable while product pages are blocked.

How to repair safely

Identify whether the behavior originates in Shopify, Theme code, a security/CDN service, or an installed app. Change one layer at a time. If editing robots.txt.liquid, duplicate the Theme and preserve Shopify’s required directives rather than replacing the file with a generic robots template.

Rollback

This guide is read-only until a configuration repair is made. Save the prior Theme file or security rule, publish one change, rerun the same commands, and restore the prior configuration if normal browser or search access changes unexpectedly.