Instagram / robots.txt
← back to instagram.com · change detected 30d ago (2026-10-07T21:56:40Z) · modified +77 −21
What changed
- 🤖 AIUnblocked Amazonbot (Amazon)
- 🤖 AIAdded Allow: /places/c/ for Amazonbot
- 🤖 AIAdded Disallow: /accounts/login/*?next= for Amazonbot
- 🤖 AIAdded Disallow: /ajax/ for Amazonbot
- 🤖 AIAdded Disallow: /client_error/ for Amazonbot
- 🤖 AIAdded Disallow: /direct/ for Amazonbot
- 🤖 AIAdded Disallow: /logging/ for Amazonbot
- 🤖 AIAdded Disallow: /publicapi/ for Amazonbot
- 🤖 AIAdded Disallow: /qp/batch_fetch_web/ for Amazonbot
- 🤖 AIAdded Disallow: /query/ for Amazonbot
- 🤖 AIUnblocked Applebot-Extended (Apple)
- 🤖 AIAdded Allow: /places/c/ for Applebot-Extended
- 🤖 AIAdded Disallow: /accounts/login/*?next= for Applebot-Extended
- 🤖 AIAdded Disallow: /ajax/ for Applebot-Extended
- 🤖 AIAdded Disallow: /client_error/ for Applebot-Extended
- 🤖 AIAdded Disallow: /direct/ for Applebot-Extended
- 🤖 AIAdded Disallow: /logging/ for Applebot-Extended
- 🤖 AIAdded Disallow: /publicapi/ for Applebot-Extended
- 🤖 AIAdded Disallow: /qp/batch_fetch_web/ for Applebot-Extended
- 🤖 AIAdded Disallow: /query/ for Applebot-Extended
- Unblocked user-agent ‘Brightbot’
- Added Allow: /places/c/ for Brightbot
- Added Disallow: /accounts/login/*?next= for Brightbot
- Added Disallow: /ajax/ for Brightbot
- Added Disallow: /client_error/ for Brightbot
- Added Disallow: /direct/ for Brightbot
- Added Disallow: /logging/ for Brightbot
- Added Disallow: /publicapi/ for Brightbot
- Added Disallow: /qp/batch_fetch_web/ for Brightbot
- Added Disallow: /query/ for Brightbot
- 🤖 AIUnblocked ClaudeBot (Anthropic)
- 🤖 AIAdded Allow: /places/c/ for ClaudeBot
- 🤖 AIAdded Disallow: /accounts/login/*?next= for ClaudeBot
- 🤖 AIAdded Disallow: /ajax/ for ClaudeBot
- 🤖 AIAdded Disallow: /client_error/ for ClaudeBot
- 🤖 AIAdded Disallow: /direct/ for ClaudeBot
- 🤖 AIAdded Disallow: /logging/ for ClaudeBot
- 🤖 AIAdded Disallow: /publicapi/ for ClaudeBot
- 🤖 AIAdded Disallow: /qp/batch_fetch_web/ for ClaudeBot
- 🤖 AIAdded Disallow: /query/ for ClaudeBot
- 🤖 AIUnblocked Google-Extended (Google)
- 🤖 AIAdded Allow: /places/c/ for Google-Extended
- 🤖 AIAdded Disallow: /accounts/login/*?next= for Google-Extended
- 🤖 AIAdded Disallow: /ajax/ for Google-Extended
- 🤖 AIAdded Disallow: /client_error/ for Google-Extended
- 🤖 AIAdded Disallow: /direct/ for Google-Extended
- 🤖 AIAdded Disallow: /logging/ for Google-Extended
- 🤖 AIAdded Disallow: /publicapi/ for Google-Extended
- 🤖 AIAdded Disallow: /qp/batch_fetch_web/ for Google-Extended
- 🤖 AIAdded Disallow: /query/ for Google-Extended
- ... and 20 more change(s)
Diff old (2026-09-23T10:50:20Z) → new (2026-10-07T21:56:40Z)
| @@ -5,27 +5,6 @@ | ||
| 5 | 5 | # All authorized user-agents listed on this page must comply with Meta’s |
| 6 | 6 | # Automated Data Collection Terms available at: |
| 7 | 7 | # https://www.facebook.com/legal/automated_data_collection_terms |
| 8 | - User-agent: Amazonbot | |
| 9 | - Disallow: / | |
| 10 | - | |
| 11 | - User-agent: Applebot-Extended | |
| 12 | - Disallow: / | |
| 13 | - | |
| 14 | - User-agent: Brightbot | |
| 15 | - Disallow: / | |
| 16 | - | |
| 17 | - User-agent: ClaudeBot | |
| 18 | - Disallow: / | |
| 19 | - | |
| 20 | - User-agent: Google-Extended | |
| 21 | - Disallow: / | |
| 22 | - | |
| 23 | - User-agent: GPTBot | |
| 24 | - Disallow: / | |
| 25 | - | |
| 26 | - User-agent: PerplexityBot | |
| 27 | - Disallow: / | |
| 28 | - | |
| 29 | 8 | User-agent: PetalBot |
| 30 | 9 | Disallow: / |
| 31 | 10 | |
| @@ -284,6 +263,83 @@ | ||
| 284 | 263 | Disallow: /query/ |
| 285 | 264 | Allow: /places/c/ |
| 286 | 265 | |
| 266 | + User-agent: Amazonbot | |
| 267 | + Disallow: /accounts/login/*?next= | |
| 268 | + Disallow: /ajax/ | |
| 269 | + Disallow: /client_error/ | |
| 270 | + Disallow: /direct/ | |
| 271 | + Disallow: /logging/ | |
| 272 | + Disallow: /publicapi/ | |
| 273 | + Disallow: /qp/batch_fetch_web/ | |
| 274 | + Disallow: /query/ | |
| 275 | + Allow: /places/c/ | |
| 276 | + | |
| 277 | + User-agent: Applebot-Extended | |
| 278 | + Disallow: /accounts/login/*?next= | |
| 279 | + Disallow: /ajax/ | |
| 280 | + Disallow: /client_error/ | |
| 281 | + Disallow: /direct/ | |
| 282 | + Disallow: /logging/ | |
| 283 | + Disallow: /publicapi/ | |
| 284 | + Disallow: /qp/batch_fetch_web/ | |
| 285 | + Disallow: /query/ | |
| 286 | + Allow: /places/c/ | |
| 287 | + | |
| 288 | + User-agent: Brightbot | |
| 289 | + Disallow: /accounts/login/*?next= | |
| 290 | + Disallow: /ajax/ | |
| 291 | + Disallow: /client_error/ | |
| 292 | + Disallow: /direct/ | |
| 293 | + Disallow: /logging/ | |
| 294 | + Disallow: /publicapi/ | |
| 295 | + Disallow: /qp/batch_fetch_web/ | |
| 296 | + Disallow: /query/ | |
| 297 | + Allow: /places/c/ | |
| 298 | + | |
| 299 | + User-agent: ClaudeBot | |
| 300 | + Disallow: /accounts/login/*?next= | |
| 301 | + Disallow: /ajax/ | |
| 302 | + Disallow: /client_error/ | |
| 303 | + Disallow: /direct/ | |
| 304 | + Disallow: /logging/ | |
| 305 | + Disallow: /publicapi/ | |
| 306 | + Disallow: /qp/batch_fetch_web/ | |
| 307 | + Disallow: /query/ | |
| 308 | + Allow: /places/c/ | |
| 309 | + | |
| 310 | + User-agent: Google-Extended | |
| 311 | + Disallow: /accounts/login/*?next= | |
| 312 | + Disallow: /ajax/ | |
| 313 | + Disallow: /client_error/ | |
| 314 | + Disallow: /direct/ | |
| 315 | + Disallow: /logging/ | |
| 316 | + Disallow: /publicapi/ | |
| 317 | + Disallow: /qp/batch_fetch_web/ | |
| 318 | + Disallow: /query/ | |
| 319 | + Allow: /places/c/ | |
| 320 | + | |
| 321 | + User-agent: GPTBot | |
| 322 | + Disallow: /accounts/login/*?next= | |
| 323 | + Disallow: /ajax/ | |
| 324 | + Disallow: /client_error/ | |
| 325 | + Disallow: /direct/ | |
| 326 | + Disallow: /logging/ | |
| 327 | + Disallow: /publicapi/ | |
| 328 | + Disallow: /qp/batch_fetch_web/ | |
| 329 | + Disallow: /query/ | |
| 330 | + Allow: /places/c/ | |
| 331 | + | |
| 332 | + User-agent: PerplexityBot | |
| 333 | + Disallow: /accounts/login/*?next= | |
| 334 | + Disallow: /ajax/ | |
| 335 | + Disallow: /client_error/ | |
| 336 | + Disallow: /direct/ | |
| 337 | + Disallow: /logging/ | |
| 338 | + Disallow: /publicapi/ | |
| 339 | + Disallow: /qp/batch_fetch_web/ | |
| 340 | + Disallow: /query/ | |
| 341 | + Allow: /places/c/ | |
| 342 | + | |
| 287 | 343 | User-agent: * |
| 288 | 344 | Disallow: / |
| 289 | 345 |