Diffbot / robots.txt
← back to diffbot.com · change detected 6d ago (2026-07-29T22:40:57Z) · modified +9 −4
What changed
- Added rules for user-agent ‘*’
- Added sitemap https://www.diffbot.com/sitemap.xml
Diff old (2026-06-20T14:56:27Z) → new (2026-07-29T22:40:57Z)
| @@ -1,10 +1,15 @@ | ||
| 1 | - # ____ _ _ _ _ | |
| 2 | - # ||o o| ___| |_ ___ _ __ _ __ ___ | |__ ___ | |_ __ _| |__ _ _ ___ ___ ___ ___ _ __ ___ | |
| 3 | - # ||===| / __| __/ _ \| '_ \| '__/ _ \| '_ \ / _ \| __/ _` | '_ \| | | / __|/ _ \ / __/ _ \| '_ ` _ \ | |
| 1 | + # ____ _ _ _ _ | |
| 2 | + # ||o o| ___| |_ ___ _ __ _ __ ___ | |__ ___ | |_ __ _| |__ _ _ ___ ___ ___ ___ _ __ ___ | |
| 3 | + # ||===| / __| __/ _ \| '_ \| '__/ _ \| '_ \ / _ \| __/ _` | '_ \| | | / __|/ _ \ / __/ _ \| '_ ` _ \ | |
| 4 | 4 | # .-.`---'-. \__ \ || (_) | |_) | | | (_) | |_) | (_) | || (_| | |_) | |_| \__ \ __/| (_| (_) | | | | | | |
| 5 | 5 | # | | o .o | |___/\__\___/| .__/|_| \___/|_.__/ \___/ \__\__,_|_.__/ \__,_|___/\___(_)___\___/|_| |_| |_| |
| 6 | - # | | o:.o | |_| | |
| 6 | + # | | o:.o | |_| | |
| 7 | 7 | # | | | |
| 8 | 8 | # `-".-.-.-' This website is officially robot abuse free. |
| 9 | 9 | # _| | : |_ |
| 10 | 10 | # (oOoOo)_)_) |
| 11 | + | |
| 12 | + User-agent: * | |
| 13 | + Allow: / | |
| 14 | + | |
| 15 | + Sitemap: https://www.diffbot.com/sitemap.xml |