Ford Motor / robots.txt
← back to ford.com · change detected 19d ago (2026-07-16T17:46:03Z) · modified +46 −0
What changed
- Blocked user-agent ‘amazon-kendra’ entirely
- Blocked user-agent ‘amazon-QBusiness’ entirely
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- 🤖 AIAdded rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AIAdded rules for AI crawler Applebot-Extended (Apple)
- Blocked user-agent ‘bedrockbot’ entirely
- Added rules for user-agent ‘ChatGPT Agent’
- 🤖 AIAdded rules for AI crawler ChatGPT-User (OpenAI)
- Added rules for user-agent ‘Claude-Code’
- 🤖 AIAdded rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-User (Anthropic)
- 🤖 AIAdded rules for AI crawler Claude-Web (Anthropic)
- 🤖 AIAdded rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIAdded rules for AI crawler FacebookBot (Meta)
- Added rules for user-agent ‘Gemini-Deep-Research’
- Added rules for user-agent ‘Google-CloudVertexBot’
- 🤖 AIAdded rules for AI crawler Google-Extended (Google)
- Added rules for user-agent ‘Google-Gemini-CLI’
- Added rules for user-agent ‘Google-NotebookLM’
- Added rules for user-agent ‘GoogleAgent-Mariner’
- Added rules for user-agent ‘GoogleOther’
- 🤖 AIAdded rules for AI crawler GPTBot (OpenAI)
- 🤖 AIAdded rules for AI crawler meta-externalagent (Meta)
- 🤖 AIAdded rules for AI crawler meta-externalfetcher (Meta)
- Added rules for user-agent ‘NotebookLM’
- 🤖 AIAdded rules for AI crawler OAI-SearchBot (OpenAI)
- Added rules for user-agent ‘OpenAI’
- Added rules for user-agent ‘Operator’
- 🤖 AIAdded rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIAdded rules for AI crawler PerplexityBot (Perplexity)
Diff old (2026-06-20T01:10:30Z) → new (2026-07-16T17:46:03Z)
| @@ -43,3 +43,49 @@ | ||
| 43 | 43 | User-agent: * |
| 44 | 44 | Crawl-delay: 10 |
| 45 | 45 | Allow: /inventory/ |
| 46 | + | |
| 47 | + #AI chatbots and assistants - permitted to access ONLY the llms.txt files | |
| 48 | + #(Google, OpenAI, Amazon, Meta, Anthropic, Perplexity, Apple) | |
| 49 | + #Allow is more specific than Disallow: / so only the llms.txt files are crawlable | |
| 50 | + #OpenAI (ChatGPT) | |
| 51 | + User-agent: GPTBot | |
| 52 | + User-agent: OAI-SearchBot | |
| 53 | + User-agent: ChatGPT-User | |
| 54 | + User-agent: ChatGPT Agent | |
| 55 | + User-agent: OpenAI | |
| 56 | + User-agent: Operator | |
| 57 | + #Google (Gemini, NotebookLM, Vertex AI) | |
| 58 | + User-agent: Google-Extended | |
| 59 | + User-agent: GoogleOther | |
| 60 | + User-agent: Gemini-Deep-Research | |
| 61 | + User-agent: Google-Gemini-CLI | |
| 62 | + User-agent: Google-NotebookLM | |
| 63 | + User-agent: NotebookLM | |
| 64 | + User-agent: Google-CloudVertexBot | |
| 65 | + User-agent: GoogleAgent-Mariner | |
| 66 | + #Anthropic (Claude) | |
| 67 | + User-agent: anthropic-ai | |
| 68 | + User-agent: ClaudeBot | |
| 69 | + User-agent: Claude-Web | |
| 70 | + User-agent: Claude-User | |
| 71 | + User-agent: Claude-SearchBot | |
| 72 | + User-agent: Claude-Code | |
| 73 | + #Perplexity | |
| 74 | + User-agent: PerplexityBot | |
| 75 | + User-agent: Perplexity-User | |
| 76 | + #Meta AI | |
| 77 | + User-agent: Meta-ExternalAgent | |
| 78 | + User-agent: meta-externalagent | |
| 79 | + User-agent: Meta-ExternalFetcher | |
| 80 | + User-agent: meta-externalfetcher | |
| 81 | + User-agent: FacebookBot | |
| 82 | + #Apple Intelligence | |
| 83 | + User-agent: Applebot-Extended | |
| 84 | + #Amazon (Alexa, Bedrock, Q) | |
| 85 | + User-agent: Amazonbot | |
| 86 | + User-agent: bedrockbot | |
| 87 | + User-agent: amazon-kendra | |
| 88 | + User-agent: amazon-QBusiness | |
| 89 | + Allow: /llms.txt | |
| 90 | + Allow: /inventory/llms.txt | |
| 91 | + Disallow: / |