Lithia Motors / robots.txt
← back to lithiamotors.com · change detected 19d ago (2026-10-18T13:46:13Z) · modified +70 −1
What changed
- 🤖 AIBlocked anthropic-ai (Anthropic) entirely
- Blocked user-agent ‘atlassian-bot’ entirely
- Blocked user-agent ‘AwarioRssBot’ entirely
- Blocked user-agent ‘AwarioSmartBot’ entirely
- Blocked user-agent ‘Baiduspider’ entirely
- Blocked user-agent ‘BorderxBot’ entirely
- Blocked user-agent ‘CitibotSiteCrawler’ entirely
- 🤖 AIBlocked Claude-Web (Anthropic) entirely
- 🤖 AIBlocked cohere-ai (Cohere) entirely
- Blocked user-agent ‘Cotoyogi’ entirely
- 🤖 AIBlocked Diffbot (Diffbot) entirely
- Blocked user-agent ‘FishBot’ entirely
- Blocked user-agent ‘Google-CloudVertexBot’ entirely
- Blocked user-agent ‘GoogleOther’ entirely
- Blocked user-agent ‘ICC-Crawler’ entirely
- Blocked user-agent ‘KimiBot’ entirely
- Blocked user-agent ‘magpie-crawler’ entirely
- Blocked user-agent ‘MistralAI-Training’ entirely
- Blocked user-agent ‘NavuBot’ entirely
- 🤖 AIBlocked omgili (Webz.io) entirely
- Blocked user-agent ‘PetalBot’ entirely
- Blocked user-agent ‘QualifiedBot’ entirely
- Blocked user-agent ‘SemrushBot-SWA’ entirely
- Blocked user-agent ‘WARDBot’ entirely
- Removed all rules for user-agent ‘CloudflareBrowserRenderingCrawler’
Diff old (2026-07-29T04:27:29Z) → new (2026-10-18T13:46:13Z)
| @@ -26,10 +26,12 @@ | ||
| 26 | 26 | |
| 27 | 27 | # BEGIN Cloudflare Managed content |
| 28 | 28 | |
| 29 | + | |
| 29 | 30 | User-agent: * |
| 30 | 31 | Content-Signal: search=yes,ai-train=no,use=reference |
| 31 | 32 | Allow: / |
| 32 | 33 | |
| 34 | + # Training crawlers | |
| 33 | 35 | User-agent: Amazonbot |
| 34 | 36 | Disallow: / |
| 35 | 37 | |
| @@ -45,7 +47,7 @@ | ||
| 45 | 47 | User-agent: ClaudeBot |
| 46 | 48 | Disallow: / |
| 47 | 49 | |
| 48 | - User-agent: CloudflareBrowserRenderingCrawler | |
| 50 | + User-agent: Diffbot | |
| 49 | 51 | Disallow: / |
| 50 | 52 | |
| 51 | 53 | User-agent: Google-Extended |
| @@ -54,7 +56,74 @@ | ||
| 54 | 56 | User-agent: GPTBot |
| 55 | 57 | Disallow: / |
| 56 | 58 | |
| 59 | + User-agent: omgili | |
| 60 | + Disallow: / | |
| 61 | + | |
| 62 | + User-agent: anthropic-ai | |
| 63 | + Disallow: / | |
| 64 | + | |
| 65 | + User-agent: Claude-Web | |
| 66 | + Disallow: / | |
| 67 | + | |
| 68 | + User-agent: cohere-ai | |
| 69 | + Disallow: / | |
| 70 | + | |
| 71 | + User-agent: MistralAI-Training | |
| 72 | + Disallow: / | |
| 73 | + | |
| 57 | 74 | User-agent: meta-externalagent |
| 75 | + Disallow: / | |
| 76 | + | |
| 77 | + User-agent: GoogleOther | |
| 78 | + Disallow: / | |
| 79 | + | |
| 80 | + User-agent: Baiduspider | |
| 81 | + Disallow: / | |
| 82 | + | |
| 83 | + User-agent: PetalBot | |
| 84 | + Disallow: / | |
| 85 | + | |
| 86 | + User-agent: AwarioSmartBot | |
| 87 | + User-agent: AwarioRssBot | |
| 88 | + Disallow: / | |
| 89 | + | |
| 90 | + User-agent: Google-CloudVertexBot | |
| 91 | + Disallow: / | |
| 92 | + | |
| 93 | + User-agent: QualifiedBot | |
| 94 | + Disallow: / | |
| 95 | + | |
| 96 | + User-agent: Cotoyogi | |
| 97 | + Disallow: / | |
| 98 | + | |
| 99 | + User-agent: ICC-Crawler | |
| 100 | + Disallow: / | |
| 101 | + | |
| 102 | + User-agent: atlassian-bot | |
| 103 | + Disallow: / | |
| 104 | + | |
| 105 | + User-agent: FishBot | |
| 106 | + Disallow: / | |
| 107 | + | |
| 108 | + User-agent: BorderxBot | |
| 109 | + Disallow: / | |
| 110 | + | |
| 111 | + User-agent: NavuBot | |
| 112 | + Disallow: / | |
| 113 | + | |
| 114 | + User-agent: SemrushBot-SWA | |
| 115 | + Disallow: / | |
| 116 | + | |
| 117 | + User-agent: WARDBot | |
| 118 | + Disallow: / | |
| 119 | + | |
| 120 | + User-agent: magpie-crawler | |
| 121 | + Disallow: / | |
| 122 | + | |
| 123 | + User-agent: KimiBot | |
| 124 | + Disallow: / | |
| 125 | + | |
| 126 | + User-agent: CitibotSiteCrawler | |
| 58 | 127 | Disallow: / |
| 59 | 128 | |
| 60 | 129 | # END Cloudflare Managed Content |