How to Audit Shopify robots.txt.liquid

Shopify generates /robots.txt automatically. Most stores should leave the default rules in place. Audit first; customize only when you can name the blocked crawler, affected path, intended policy, and rollback.

1. Capture the live file

curl -sS https://example.com/robots.txt | tee robots-before.txt

Confirm a 200 response and a plain-text body. Record the date because apps and theme deployments can change behavior.

2. Map priority paths

Test at least:

• homepage
• two collections
• five priority products
• one editorial guide
• sitemap
• agents.md

Cart, checkout, account, search parameters, and internal utility paths are not equivalent to product evidence pages.

3. Evaluate matching groups

For each crawler in scope:

1. find the most specific matching user-agent group
2. list its Allow and Disallow rules
3. test the exact URL path
4. inspect CDN or WAF behavior separately

Do not infer access from User-agent: * if a more specific group exists.

4. Locate customization

In Shopify admin:

Online Store → Themes → … → Edit code

Search for templates/robots.txt.liquid. Its presence means the store has overridden or extended generated output. Review theme history and app ownership before editing.

5. Preserve default groups

If customization is justified, start from Shopify’s Liquid objects:

{% for group in robots.default_groups %}
  {{- group.user_agent }}
  {% for rule in group.rules %}
    {{- rule }}
  {% endfor %}
  {%- if group.sitemap != blank %}
    {{ group.sitemap }}
  {% endif %}
{% endfor %}

Do not paste a static robots.txt copied from another store. It can omit Shopify protections and point to the wrong sitemap.

6. Test a theme preview

Fetch the preview output where possible and compare:

diff -u robots-before.txt robots-after.txt

Review every changed rule, not just the rule you intended to add.

7. Verify outside robots

curl -sSIL https://example.com/products/example-product
curl -sSIL -A 'DOCUMENTED-CRAWLER-USER-AGENT' \
  https://example.com/products/example-product

Use the provider’s current official user agent. Then confirm legitimate crawler traffic using logs and documented IP validation. A spoofed header is not proof.

Acceptance criteria

• priority URLs are not unintentionally disallowed
• sitemap is present and reachable
• generated Shopify rules remain intact
• CDN/WAF does not challenge approved traffic
• a rollback version is retained
• the policy owner and reason are documented

Reference: Shopify robots.txt.liquid documentation.