# ========================================================= # EducateToSave.com # Search-friendly AI crawler policy # ========================================================= # --------------------------------------------------------- # GENERAL CRAWLERS # Public pages may be crawled unless explicitly disallowed. # --------------------------------------------------------- User-agent: * Disallow: /application/ Disallow: /cgi-bin/ Disallow: /system/ Disallow: /logs/ Disallow: /admin Disallow: /etsadminbs Disallow: /freadmin Disallow: /mrkadmin Disallow: /orkadmin Disallow: /ttadmin Disallow: /tadmin Disallow: /tatadmin Disallow: /loeadmin Disallow: /finance Disallow: /gpsinghfin Disallow: /uksponsors Disallow: /ussponsors Disallow: /canada Disallow: /utility Disallow: /api Disallow: /AllSponsor Disallow: /allSponsor Disallow: /Student Disallow: /student Disallow: /NCLBStudents Disallow: /profile/ Disallow: /Dashboard Disallow: /dashboard Disallow: /Account Disallow: /account Disallow: /ccavenue/ Disallow: /PaypalSDK/ Disallow: /webservices/ Disallow: /sendgrid/ Disallow: /Kaleyra/ Disallow: /google_drive_file_upload_with_php/ Disallow: /Index/getNewsroomPosts Disallow: /Index/getStoriesPosts Disallow: /index/getNewsroomPosts Disallow: /index/getStoriesPosts Disallow: /dev.vars.production Disallow: /.env Disallow: /lib/.env # --------------------------------------------------------- # SEARCH ENGINES AND AI SEARCH DISCOVERY # Allow public content while excluding sensitive paths. # --------------------------------------------------------- User-agent: Googlebot User-agent: Bingbot User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Amazonbot User-agent: Applebot User-agent: Meta-ExternalFetcher Disallow: /application/ Disallow: /cgi-bin/ Disallow: /system/ Disallow: /logs/ Disallow: /admin Disallow: /etsadminbs Disallow: /freadmin Disallow: /mrkadmin Disallow: /orkadmin Disallow: /ttadmin Disallow: /tadmin Disallow: /tatadmin Disallow: /loeadmin Disallow: /finance Disallow: /gpsinghfin Disallow: /uksponsors Disallow: /ussponsors Disallow: /canada Disallow: /utility Disallow: /api Disallow: /AllSponsor Disallow: /allSponsor Disallow: /Student Disallow: /student Disallow: /NCLBStudents Disallow: /profile/ Disallow: /Dashboard Disallow: /dashboard Disallow: /Account Disallow: /account Disallow: /ccavenue/ Disallow: /PaypalSDK/ Disallow: /webservices/ Disallow: /sendgrid/ Disallow: /Kaleyra/ Disallow: /google_drive_file_upload_with_php/ Disallow: /Index/getNewsroomPosts Disallow: /Index/getStoriesPosts Disallow: /index/getNewsroomPosts Disallow: /index/getStoriesPosts Disallow: /dev.vars.production Disallow: /.env Disallow: /lib/.env # --------------------------------------------------------- # USER-INITIATED AI FETCHING # Public pages allowed; sensitive paths restricted. # robots.txt is not an access-control mechanism. # --------------------------------------------------------- User-agent: ChatGPT-User User-agent: Claude-User User-agent: Perplexity-User Disallow: /application/ Disallow: /cgi-bin/ Disallow: /system/ Disallow: /logs/ Disallow: /admin Disallow: /etsadminbs Disallow: /freadmin Disallow: /mrkadmin Disallow: /orkadmin Disallow: /ttadmin Disallow: /tadmin Disallow: /tatadmin Disallow: /loeadmin Disallow: /finance Disallow: /gpsinghfin Disallow: /uksponsors Disallow: /ussponsors Disallow: /canada Disallow: /utility Disallow: /api Disallow: /AllSponsor Disallow: /allSponsor Disallow: /Student Disallow: /student Disallow: /NCLBStudents Disallow: /profile/ Disallow: /Dashboard Disallow: /dashboard Disallow: /Account Disallow: /account Disallow: /.env Disallow: /lib/.env # --------------------------------------------------------- # AI MODEL-TRAINING CRAWLERS # Disallow crawling for training-related purposes. # --------------------------------------------------------- User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Meta-ExternalAgent Disallow: / # --------------------------------------------------------- # OTHER UNWANTED CRAWLERS # Retain the existing restrictions. # --------------------------------------------------------- User-agent: Scrapy Disallow: / User-agent: Nutch Disallow: / User-agent: Baiduspider Disallow: / User-agent: DotBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: trendictionbot Disallow: / User-agent: Yeti Disallow: / # --------------------------------------------------------- # SITEMAP # --------------------------------------------------------- Sitemap: https://educatetosave.com/sitemap.xml # AI content guide, if published: # https://educatetosave.com/llms.txt