# Block 1: Main AI training crawlers User-agent: GPTBot User-agent: ClaudeBot User-agent: PerplexityBot User-agent: OAI-SearchBot User-agent: Applebot-Extended User-agent: CoPilot # Allow them to crawl your website and your raw markdown storage folder Allow: / Allow: /ta/testsmd/ Allow: /ta/cpgmd/ Allow: /ta/blogsmd/ Allow: /ta/pressmd/ # UNIVERSAL DISALLOWS (Must repeat here so AI bots don't crawl them) Disallow: /ta/account/ Disallow: /ta/affiliate/ Disallow: /ta/analytics/ Disallow: /ta/autoreport/ Disallow: /ta/ct5/ Disallow: /ta/evalplan/ Disallow: /ta/include/ Disallow: /ta/job/ Disallow: /ta/junk/ Disallow: /ta/lvi/ Disallow: /ta/misc/adp/ Disallow: /ta/misc/av/ Disallow: /ta/misc/cledit/ Disallow: /ta/misc/custom/ Disallow: /ta/misc/dwnld/ Disallow: /ta/misc/error/ Disallow: /ta/misc/errorstatic/ Disallow: /ta/misc/idle/ Disallow: /ta/misc/ivr/ Disallow: /ta/misc/job/ Disallow: /ta/misc/lal/ Disallow: /ta/misc/offline/ Disallow: /ta/misc/proctor/ Disallow: /ta/misc/seb/ Disallow: /ta/misc/tkextend/ Disallow: /ta/misc/widget/ Disallow: /ta/order/ Disallow: /ta/ref/ Disallow: /ta/resources/ Disallow: /ta/skin/ Disallow: /ta/tools/ Disallow: /ta/user/ Disallow: /ta/WEB-INF/ # They will also read the global sitemaps at the bottom of the file # Block 2: Standard Search Engines (Google, Bing, etc.) User-agent: Googlebot User-agent: Bingbot User-agent: Yandex User-agent: Baiduspider Allow: / # CRITICAL: Prevent search engines from indexing raw markdown URLs Disallow: /ta/testsmd/ Disallow: /ta/cpgmd/ Disallow: /ta/blogsmd/ Disallow: /ta/pressmd/ # UNIVERSAL DISALLOWS (Must repeat here so AI bots don't crawl them) Disallow: /ta/account/ Disallow: /ta/affiliate/ Disallow: /ta/analytics/ Disallow: /ta/autoreport/ Disallow: /ta/ct5/ Disallow: /ta/evalplan/ Disallow: /ta/include/ Disallow: /ta/job/ Disallow: /ta/junk/ Disallow: /ta/lvi/ Disallow: /ta/misc/adp/ Disallow: /ta/misc/av/ Disallow: /ta/misc/cledit/ Disallow: /ta/misc/custom/ Disallow: /ta/misc/dwnld/ Disallow: /ta/misc/error/ Disallow: /ta/misc/errorstatic/ Disallow: /ta/misc/idle/ Disallow: /ta/misc/ivr/ Disallow: /ta/misc/job/ Disallow: /ta/misc/lal/ Disallow: /ta/misc/offline/ Disallow: /ta/misc/proctor/ Disallow: /ta/misc/seb/ Disallow: /ta/misc/tkextend/ Disallow: /ta/misc/widget/ Disallow: /ta/order/ Disallow: /ta/ref/ Disallow: /ta/resources/ Disallow: /ta/skin/ Disallow: /ta/tools/ Disallow: /ta/user/ Disallow: /ta/WEB-INF/ # 3. CATCH-ALL FOR ALL OTHER BOTS User-agent: * Allow: / # disallow markdowns Disallow: /ta/testsmd/ Disallow: /ta/cpgmd/ Disallow: /ta/blogsmd/ Disallow: /ta/pressmd/ # disallow other Disallow: /ta/account/ Disallow: /ta/affiliate/ Disallow: /ta/analytics/ Disallow: /ta/autoreport/ Disallow: /ta/ct5/ Disallow: /ta/evalplan/ Disallow: /ta/include/ Disallow: /ta/job/ Disallow: /ta/junk/ Disallow: /ta/lvi/ Disallow: /ta/misc/adp/ Disallow: /ta/misc/av/ Disallow: /ta/misc/cledit/ Disallow: /ta/misc/custom/ Disallow: /ta/misc/dwnld/ Disallow: /ta/misc/error/ Disallow: /ta/misc/errorstatic/ Disallow: /ta/misc/idle/ Disallow: /ta/misc/ivr/ Disallow: /ta/misc/job/ Disallow: /ta/misc/lal/ Disallow: /ta/misc/offline/ Disallow: /ta/misc/proctor/ Disallow: /ta/misc/seb/ Disallow: /ta/misc/tkextend/ Disallow: /ta/misc/widget/ Disallow: /ta/order/ Disallow: /ta/ref/ Disallow: /ta/resources/ Disallow: /ta/skin/ Disallow: /ta/tools/ Disallow: /ta/user/ Disallow: /ta/WEB-INF/ # 4. GLOBAL SITEMAP DIRECTIVES (Read by ALL bots) # Standard search engines will attempt to read both, but will skip the AI one # because they are explicitly disallowed from crawling the .md URLs inside it. Sitemap: https://www.hravatar.com/sitemap.xml Sitemap: https://www.hravatar.com/sitemap-html.xml Sitemap: https://www.hravatar.com/sitemap-md.xml