- Response is a valid robots.txt again (was an HTML page)
- Added rules for user-agent ‘*’
- 🤖 AI Blocked AI2Bot (Allen Institute for AI) entirely
- Blocked user-agent ‘AI2Bot-Dolma’ entirely
- 🤖 AI Blocked Amazonbot (Amazon) entirely
- 🤖 AI Blocked anthropic-ai (Anthropic) entirely
- … 41 more change(s)
The Pulse
Every change to a monitored robots.txt, archived and explained.
1273sites tracked
1206files archived
5420snapshots
4284changes detected
241this week
- Response is a valid robots.txt again (was an HTML page)
- Added rules for user-agent ‘*’
- 🤖 AI Blocked GPTBot (OpenAI) entirely
- Added sitemap https://www.khanacademy.org/sitemap.xml
- Response is a valid robots.txt again (was an HTML page)
- Added rules for user-agent ‘*’
- Blocked user-agent ‘facebookcatalog’ entirely
- Blocked user-agent ‘facebookexternalhit’ entirely
- 🤖 AI Blocked meta-externalagent (Meta) entirely