# robots.txt for PsychicWho # https://psychicwho.com/robots.txt # Content-Signal (https://contentsignals.org): search indexing and use as AI answer # input (with citation) are welcome; AI model training is not. It matches the bot # groups below and, like Disallow, has to be repeated in every named group. # Standard crawlers # No Crawl-delay: Google ignores the directive and Bing throttles to 1 req/s for # every URL, which slows discovery of a 2,400-URL directory for no benefit. User-agent: * Allow: / Disallow: /go/ Content-Signal: search=yes, ai-input=yes, ai-train=no # --- AI Search / Citation bots (ALLOW) --- # These cite us in AI responses — direct traffic benefit. # A crawler obeys ONLY its most specific User-agent group and ignores `*`, so # every named group must repeat `Disallow: /go/` or that bot may fetch the # outbound redirects (each fetch used to count as an affiliate click). # Anthropic (Claude search) User-agent: ClaudeBot Allow: / Disallow: /go/ Content-Signal: search=yes, ai-input=yes, ai-train=no # OpenAI search (ChatGPT citation — NOT training) User-agent: OAI-SearchBot Allow: / Disallow: /go/ Content-Signal: search=yes, ai-input=yes, ai-train=no User-agent: ChatGPT-User Allow: / Disallow: /go/ Content-Signal: search=yes, ai-input=yes, ai-train=no # Perplexity User-agent: PerplexityBot Allow: / Disallow: /go/ Content-Signal: search=yes, ai-input=yes, ai-train=no # Apple User-agent: Applebot Allow: / Disallow: /go/ Content-Signal: search=yes, ai-input=yes, ai-train=no # --- AI Training bots (DISALLOW) --- # These only collect training data — no citation benefit. # OpenAI training crawler User-agent: GPTBot Disallow: / # Google Gemini training (does NOT affect Google Search or AI Overviews) User-agent: Google-Extended Disallow: / # Apple Intelligence training User-agent: Applebot-Extended Disallow: / # Common Crawl training dataset User-agent: CCBot Disallow: / # ByteDance — high bandwidth, no value User-agent: Bytespider Disallow: / # Meta AI training User-agent: Meta-ExternalAgent Disallow: / # Sitemap Sitemap: https://psychicwho.com/sitemap-index.xml