American Financial Group / robots.txt
← back to afginc.com · change detected 9d ago (2026-06-17T01:55:34Z) · modified +65 −2036
What changed
- Response is a valid robots.txt again (was an HTML page)
- Added rules for user-agent ‘*’
- Blocked user-agent ‘facebookcatalog’ entirely
- Blocked user-agent ‘facebookexternalhit’ entirely
- 🤖 AIBlocked meta-externalagent (Meta) entirely
Diff (no previous version) → (file removed)
No line-level differences (identical or both empty).