Monitor Shopify Crawler Access After Installing an App

Baseline before install

Check the homepage, one collection, one simple product, one multi-variant product, and one sold-out product.

mkdir -p baseline
for path in / /collections/all /products/example; do
  name=$(printf '%s' "$path" | tr '/?' '__')
  curl -sSL -D "baseline/$name.headers" -o "baseline/$name.html" "https://example.com$path"
done
curl -sS https://example.com/robots.txt > baseline/robots.txt

Record status, final URL, body size, canonical, Product block count, and challenge signatures.

After install

Repeat into after/ and compare semantic fields, not the entire minified HTML.

rg -n 'canonical|application/ld\+json|availability|priceCurrency' baseline/*.html > baseline-signals.txt
rg -n 'canonical|application/ld\+json|availability|priceCurrency' after/*.html > after-signals.txt
diff -u baseline-signals.txt after-signals.txt

Real failure classes

• SEO app adds a second Product with stale stock
• review app emits zero aggregate before its widget loads
• security app challenges bot-like requests
• localization app changes market redirect and canonical
• subscription app replaces one-time price in schema

The Woobles conflict demonstrates why duplicate owners matter; Ring’s explicit public/private robots rules demonstrate why broad allow/deny changes need review.

User-agent comparison

Repeat approved response tests for OAI-SearchBot, PerplexityBot, Claude-SearchBot, and Googlebot strings, while treating strings as spoofable diagnostics.

Rollback

Disable the app embed in a duplicate theme first. If the app changes CDN or account-level settings, retain the previous rule and app configuration before removal.

Acceptance criteria

No unintended status, redirect, canonical, robots, schema-owner, price, or stock change appears across fixtures. Any intentional change has an owner and rollback.