Instagram / robots.txt
← back to instagram.com · change detected 19d ago (2026-10-18T13:46:13Z) · modified +18 −66
What changed
- 🤖 AIBlocked Amazonbot (Amazon) entirely
- 🤖 AIRemoved Allow: /places/c/ for Amazonbot
- 🤖 AIRemoved Disallow: /accounts/login/*?next= for Amazonbot
- 🤖 AIRemoved Disallow: /ajax/ for Amazonbot
- 🤖 AIRemoved Disallow: /client_error/ for Amazonbot
- 🤖 AIRemoved Disallow: /direct/ for Amazonbot
- 🤖 AIRemoved Disallow: /logging/ for Amazonbot
- 🤖 AIRemoved Disallow: /publicapi/ for Amazonbot
- 🤖 AIRemoved Disallow: /qp/batch_fetch_web/ for Amazonbot
- 🤖 AIRemoved Disallow: /query/ for Amazonbot
- 🤖 AIBlocked Applebot-Extended (Apple) entirely
- 🤖 AIRemoved Allow: /places/c/ for Applebot-Extended
- 🤖 AIRemoved Disallow: /accounts/login/*?next= for Applebot-Extended
- 🤖 AIRemoved Disallow: /ajax/ for Applebot-Extended
- 🤖 AIRemoved Disallow: /client_error/ for Applebot-Extended
- 🤖 AIRemoved Disallow: /direct/ for Applebot-Extended
- 🤖 AIRemoved Disallow: /logging/ for Applebot-Extended
- 🤖 AIRemoved Disallow: /publicapi/ for Applebot-Extended
- 🤖 AIRemoved Disallow: /qp/batch_fetch_web/ for Applebot-Extended
- 🤖 AIRemoved Disallow: /query/ for Applebot-Extended
- Blocked user-agent ‘Brightbot’ entirely
- Removed Allow: /places/c/ for Brightbot
- Removed Disallow: /accounts/login/*?next= for Brightbot
- Removed Disallow: /ajax/ for Brightbot
- Removed Disallow: /client_error/ for Brightbot
- Removed Disallow: /direct/ for Brightbot
- Removed Disallow: /logging/ for Brightbot
- Removed Disallow: /publicapi/ for Brightbot
- Removed Disallow: /qp/batch_fetch_web/ for Brightbot
- Removed Disallow: /query/ for Brightbot
- 🤖 AIBlocked ClaudeBot (Anthropic) entirely
- 🤖 AIRemoved Allow: /places/c/ for ClaudeBot
- 🤖 AIRemoved Disallow: /accounts/login/*?next= for ClaudeBot
- 🤖 AIRemoved Disallow: /ajax/ for ClaudeBot
- 🤖 AIRemoved Disallow: /client_error/ for ClaudeBot
- 🤖 AIRemoved Disallow: /direct/ for ClaudeBot
- 🤖 AIRemoved Disallow: /logging/ for ClaudeBot
- 🤖 AIRemoved Disallow: /publicapi/ for ClaudeBot
- 🤖 AIRemoved Disallow: /qp/batch_fetch_web/ for ClaudeBot
- 🤖 AIRemoved Disallow: /query/ for ClaudeBot
- 🤖 AIBlocked GPTBot (OpenAI) entirely
- 🤖 AIRemoved Allow: /places/c/ for GPTBot
- 🤖 AIRemoved Disallow: /accounts/login/*?next= for GPTBot
- 🤖 AIRemoved Disallow: /ajax/ for GPTBot
- 🤖 AIRemoved Disallow: /client_error/ for GPTBot
- 🤖 AIRemoved Disallow: /direct/ for GPTBot
- 🤖 AIRemoved Disallow: /logging/ for GPTBot
- 🤖 AIRemoved Disallow: /publicapi/ for GPTBot
- 🤖 AIRemoved Disallow: /qp/batch_fetch_web/ for GPTBot
- 🤖 AIRemoved Disallow: /query/ for GPTBot
- ... and 10 more change(s)
Diff old (2026-10-15T06:25:59Z) → new (2026-10-18T13:46:13Z)
| @@ -5,7 +5,25 @@ | ||
| 5 | 5 | # All authorized user-agents listed on this page must comply with Meta’s |
| 6 | 6 | # Automated Data Collection Terms available at: |
| 7 | 7 | # https://www.facebook.com/legal/automated_data_collection_terms |
| 8 | + User-agent: Amazonbot | |
| 9 | + Disallow: / | |
| 10 | + | |
| 11 | + User-agent: Applebot-Extended | |
| 12 | + Disallow: / | |
| 13 | + | |
| 14 | + User-agent: Brightbot | |
| 15 | + Disallow: / | |
| 16 | + | |
| 17 | + User-agent: ClaudeBot | |
| 18 | + Disallow: / | |
| 19 | + | |
| 8 | 20 | User-agent: Google-Extended |
| 21 | + Disallow: / | |
| 22 | + | |
| 23 | + User-agent: GPTBot | |
| 24 | + Disallow: / | |
| 25 | + | |
| 26 | + User-agent: PerplexityBot | |
| 9 | 27 | Disallow: / |
| 10 | 28 | |
| 11 | 29 | User-agent: PetalBot |
| @@ -266,72 +284,6 @@ | ||
| 266 | 284 | Disallow: /query/ |
| 267 | 285 | Allow: /places/c/ |
| 268 | 286 | |
| 269 | - User-agent: Amazonbot | |
| 270 | - Disallow: /accounts/login/*?next= | |
| 271 | - Disallow: /ajax/ | |
| 272 | - Disallow: /client_error/ | |
| 273 | - Disallow: /direct/ | |
| 274 | - Disallow: /logging/ | |
| 275 | - Disallow: /publicapi/ | |
| 276 | - Disallow: /qp/batch_fetch_web/ | |
| 277 | - Disallow: /query/ | |
| 278 | - Allow: /places/c/ | |
| 279 | - | |
| 280 | - User-agent: Applebot-Extended | |
| 281 | - Disallow: /accounts/login/*?next= | |
| 282 | - Disallow: /ajax/ | |
| 283 | - Disallow: /client_error/ | |
| 284 | - Disallow: /direct/ | |
| 285 | - Disallow: /logging/ | |
| 286 | - Disallow: /publicapi/ | |
| 287 | - Disallow: /qp/batch_fetch_web/ | |
| 288 | - Disallow: /query/ | |
| 289 | - Allow: /places/c/ | |
| 290 | - | |
| 291 | - User-agent: Brightbot | |
| 292 | - Disallow: /accounts/login/*?next= | |
| 293 | - Disallow: /ajax/ | |
| 294 | - Disallow: /client_error/ | |
| 295 | - Disallow: /direct/ | |
| 296 | - Disallow: /logging/ | |
| 297 | - Disallow: /publicapi/ | |
| 298 | - Disallow: /qp/batch_fetch_web/ | |
| 299 | - Disallow: /query/ | |
| 300 | - Allow: /places/c/ | |
| 301 | - | |
| 302 | - User-agent: ClaudeBot | |
| 303 | - Disallow: /accounts/login/*?next= | |
| 304 | - Disallow: /ajax/ | |
| 305 | - Disallow: /client_error/ | |
| 306 | - Disallow: /direct/ | |
| 307 | - Disallow: /logging/ | |
| 308 | - Disallow: /publicapi/ | |
| 309 | - Disallow: /qp/batch_fetch_web/ | |
| 310 | - Disallow: /query/ | |
| 311 | - Allow: /places/c/ | |
| 312 | - | |
| 313 | - User-agent: GPTBot | |
| 314 | - Disallow: /accounts/login/*?next= | |
| 315 | - Disallow: /ajax/ | |
| 316 | - Disallow: /client_error/ | |
| 317 | - Disallow: /direct/ | |
| 318 | - Disallow: /logging/ | |
| 319 | - Disallow: /publicapi/ | |
| 320 | - Disallow: /qp/batch_fetch_web/ | |
| 321 | - Disallow: /query/ | |
| 322 | - Allow: /places/c/ | |
| 323 | - | |
| 324 | - User-agent: PerplexityBot | |
| 325 | - Disallow: /accounts/login/*?next= | |
| 326 | - Disallow: /ajax/ | |
| 327 | - Disallow: /client_error/ | |
| 328 | - Disallow: /direct/ | |
| 329 | - Disallow: /logging/ | |
| 330 | - Disallow: /publicapi/ | |
| 331 | - Disallow: /qp/batch_fetch_web/ | |
| 332 | - Disallow: /query/ | |
| 333 | - Allow: /places/c/ | |
| 334 | - | |
| 335 | 287 | User-agent: * |
| 336 | 288 | Disallow: / |
| 337 | 289 |