Ford Motor / robots.txt
← back to ford.com · change detected 17d ago (2026-07-18T00:21:17Z) · modified +2 −46
What changed
- Removed all rules for user-agent ‘amazon-kendra’
- Removed all rules for user-agent ‘amazon-QBusiness’
- 🤖 AIUnblocked Amazonbot (Amazon) - rules removed
- 🤖 AIRemoved rules for AI crawler anthropic-ai (Anthropic)
- 🤖 AIRemoved rules for AI crawler Applebot-Extended (Apple)
- Removed all rules for user-agent ‘bedrockbot’
- Removed all rules for user-agent ‘ChatGPT Agent’
- 🤖 AIRemoved rules for AI crawler ChatGPT-User (OpenAI)
- Removed all rules for user-agent ‘Claude-Code’
- 🤖 AIRemoved rules for AI crawler Claude-SearchBot (Anthropic)
- 🤖 AIRemoved rules for AI crawler Claude-User (Anthropic)
- 🤖 AIRemoved rules for AI crawler Claude-Web (Anthropic)
- 🤖 AIRemoved rules for AI crawler ClaudeBot (Anthropic)
- 🤖 AIRemoved rules for AI crawler FacebookBot (Meta)
- Removed all rules for user-agent ‘Gemini-Deep-Research’
- Removed all rules for user-agent ‘Google-CloudVertexBot’
- 🤖 AIRemoved rules for AI crawler Google-Extended (Google)
- Removed all rules for user-agent ‘Google-Gemini-CLI’
- Removed all rules for user-agent ‘Google-NotebookLM’
- Removed all rules for user-agent ‘GoogleAgent-Mariner’
- Removed all rules for user-agent ‘GoogleOther’
- 🤖 AIRemoved rules for AI crawler GPTBot (OpenAI)
- 🤖 AIRemoved rules for AI crawler meta-externalagent (Meta)
- 🤖 AIRemoved rules for AI crawler meta-externalfetcher (Meta)
- Removed all rules for user-agent ‘NotebookLM’
- 🤖 AIRemoved rules for AI crawler OAI-SearchBot (OpenAI)
- Removed all rules for user-agent ‘OpenAI’
- Removed all rules for user-agent ‘Operator’
- 🤖 AIRemoved rules for AI crawler Perplexity-User (Perplexity)
- 🤖 AIRemoved rules for AI crawler PerplexityBot (Perplexity)
- Added Allow: /inventory/llms.txt for *
- Added Allow: /llms.txt for *
Diff old (2026-07-16T17:46:03Z) → new (2026-07-18T00:21:17Z)
| @@ -1,5 +1,7 @@ | ||
| 1 | 1 | #robots.txt for www.Ford.com/es.ford.com - NS - 20231106 |
| 2 | 2 | User-agent: * |
| 3 | + Allow: /llms.txt | |
| 4 | + Allow: /inventory/llms.txt | |
| 3 | 5 | Disallow: /content/experience-fragments/na/* |
| 4 | 6 | Disallow: /search/ |
| 5 | 7 | Disallow: /trucks/ranger/index.amp/ |
| @@ -43,49 +45,3 @@ | ||
| 43 | 45 | User-agent: * |
| 44 | 46 | Crawl-delay: 10 |
| 45 | 47 | Allow: /inventory/ |
| 46 | - | |
| 47 | - #AI chatbots and assistants - permitted to access ONLY the llms.txt files | |
| 48 | - #(Google, OpenAI, Amazon, Meta, Anthropic, Perplexity, Apple) | |
| 49 | - #Allow is more specific than Disallow: / so only the llms.txt files are crawlable | |
| 50 | - #OpenAI (ChatGPT) | |
| 51 | - User-agent: GPTBot | |
| 52 | - User-agent: OAI-SearchBot | |
| 53 | - User-agent: ChatGPT-User | |
| 54 | - User-agent: ChatGPT Agent | |
| 55 | - User-agent: OpenAI | |
| 56 | - User-agent: Operator | |
| 57 | - #Google (Gemini, NotebookLM, Vertex AI) | |
| 58 | - User-agent: Google-Extended | |
| 59 | - User-agent: GoogleOther | |
| 60 | - User-agent: Gemini-Deep-Research | |
| 61 | - User-agent: Google-Gemini-CLI | |
| 62 | - User-agent: Google-NotebookLM | |
| 63 | - User-agent: NotebookLM | |
| 64 | - User-agent: Google-CloudVertexBot | |
| 65 | - User-agent: GoogleAgent-Mariner | |
| 66 | - #Anthropic (Claude) | |
| 67 | - User-agent: anthropic-ai | |
| 68 | - User-agent: ClaudeBot | |
| 69 | - User-agent: Claude-Web | |
| 70 | - User-agent: Claude-User | |
| 71 | - User-agent: Claude-SearchBot | |
| 72 | - User-agent: Claude-Code | |
| 73 | - #Perplexity | |
| 74 | - User-agent: PerplexityBot | |
| 75 | - User-agent: Perplexity-User | |
| 76 | - #Meta AI | |
| 77 | - User-agent: Meta-ExternalAgent | |
| 78 | - User-agent: meta-externalagent | |
| 79 | - User-agent: Meta-ExternalFetcher | |
| 80 | - User-agent: meta-externalfetcher | |
| 81 | - User-agent: FacebookBot | |
| 82 | - #Apple Intelligence | |
| 83 | - User-agent: Applebot-Extended | |
| 84 | - #Amazon (Alexa, Bedrock, Q) | |
| 85 | - User-agent: Amazonbot | |
| 86 | - User-agent: bedrockbot | |
| 87 | - User-agent: amazon-kendra | |
| 88 | - User-agent: amazon-QBusiness | |
| 89 | - Allow: /llms.txt | |
| 90 | - Allow: /inventory/llms.txt | |
| 91 | - Disallow: / |