User-agent: * Content-Signal: search=yes, ai-train=yes, ai-input=yes Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /*share= Disallow: /*?_gl= Disallow: */feed/ Disallow: /tag/ Disallow: /search/*upage= # /search/ itself is deliberately crawlable. The Search page (ID 165460) is # noindex, and Google must be able to fetch it to see that; blocking it would # entrench any already-indexed search URLs. Only deep search pagination is # blocked (Semrush 6 Aug: /search/?upage=20 and ?upage=24 timed out on crawl). # Note /*upage= must NOT be blocked globally — hub archive pagination needs it. # Re-block /search/ entirely once GSC reports no /search/ URLs indexed. # AI crawlers - full access to content, but NOT to filter permutations. # A matching user-agent group replaces the "*" group entirely and does # not inherit from it, so these rules must be repeated here. User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent Disallow: /wp-admin/ Disallow: /*share= Disallow: /*?_gl= Disallow: */feed/ Disallow: /tag/ Disallow: /search/*upage= Allow: / Sitemap: https://price-target.com/sitemap_index.xml Sitemap: https://price-target.com/sitemap-news.xml