Instagram / robots.txt
← back to instagram.com · change detected 23d ago (2026-10-15T06:25:59Z) · modified +3 −11
What changed
- 🤖 AIBlocked Google-Extended (Google) entirely
- 🤖 AIRemoved Allow: /places/c/ for Google-Extended
- 🤖 AIRemoved Disallow: /accounts/login/*?next= for Google-Extended
- 🤖 AIRemoved Disallow: /ajax/ for Google-Extended
- 🤖 AIRemoved Disallow: /client_error/ for Google-Extended
- 🤖 AIRemoved Disallow: /direct/ for Google-Extended
- 🤖 AIRemoved Disallow: /logging/ for Google-Extended
- 🤖 AIRemoved Disallow: /publicapi/ for Google-Extended
- 🤖 AIRemoved Disallow: /qp/batch_fetch_web/ for Google-Extended
- 🤖 AIRemoved Disallow: /query/ for Google-Extended
Diff old (2026-10-07T21:56:40Z) → new (2026-10-15T06:25:59Z)
| @@ -5,6 +5,9 @@ | ||
| 5 | 5 | # All authorized user-agents listed on this page must comply with Meta’s |
| 6 | 6 | # Automated Data Collection Terms available at: |
| 7 | 7 | # https://www.facebook.com/legal/automated_data_collection_terms |
| 8 | + User-agent: Google-Extended | |
| 9 | + Disallow: / | |
| 10 | + | |
| 8 | 11 | User-agent: PetalBot |
| 9 | 12 | Disallow: / |
| 10 | 13 | |
| @@ -307,17 +310,6 @@ | ||
| 307 | 310 | Disallow: /query/ |
| 308 | 311 | Allow: /places/c/ |
| 309 | 312 | |
| 310 | - User-agent: Google-Extended | |
| 311 | - Disallow: /accounts/login/*?next= | |
| 312 | - Disallow: /ajax/ | |
| 313 | - Disallow: /client_error/ | |
| 314 | - Disallow: /direct/ | |
| 315 | - Disallow: /logging/ | |
| 316 | - Disallow: /publicapi/ | |
| 317 | - Disallow: /qp/batch_fetch_web/ | |
| 318 | - Disallow: /query/ | |
| 319 | - Allow: /places/c/ | |
| 320 | - | |
| 321 | 313 | User-agent: GPTBot |
| 322 | 314 | Disallow: /accounts/login/*?next= |
| 323 | 315 | Disallow: /ajax/ |