NeuralCrawl

πŸ‡ΊπŸ‡Έ Microsoft Bing

bing.com · SEO & AI search · rank #11 · Search engine · live robots.txt ↗

AI crawler access (latest snapshot, 4h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 2570 bytes · sha256 9eae498db200 · raw

User-agent: msnbot-media
Disallow: /
Allow: /th?

User-agent: Twitterbot
Disallow:

User-agent: *
Disallow: /academic/profile
Disallow: /academic/search
Disallow: /account/
Disallow: /aclick
Disallow: /alink
Disallow: /amp/
Allow: /api/maps/
Disallow: /api/
Disallow: /bfp/search
Disallow: /bing-site-safety
Disallow: /blogs/search/
Disallow: /ck/
Disallow: /copilotsearch?q=
Disallow: /cr$
Disallow: /cr?
Disallow: /entities/search
Disallow: /entityexplore$
Disallow: /entityexplore?
Disallow: /fd/
Disallow: /fun/api/
Disallow: /fun/g/
Disallow: /history
Disallow: /hotel/reviews?
Disallow: /hotels/search
Disallow: /HpImageArchive.aspx
Disallow: /hpm
Disallow: /hpmob
Disallow: /images/async/?
Disallow: /images/async?
Disallow: /images/hosted/?
Disallow: /images/hosted/search?
Disallow: /images/hosted?
Disallow: /images/search%3F
Disallow: /images/search%5C
Disallow: /images/search/?
Disallow: /images/search?
Disallow: /images/searchbyimage
Disallow: /images?
Disallow: /inboxcommerce
Disallow: /local
Disallow: /maps/adsendpoint
Disallow: /merchant/reviews?
Disallow: /notifications/
Disallow: /offers/proxy/dealsserver/api/log
Disallow: /offers/proxy/dealsserver/buy
Disallow: /ping
Disallow: /product/reviews?
Disallow: /profile/history?
Disallow: /proFile/history?
Disallow: /Proxy.ashx
Disallow: /results
Disallow: /rewardsapp/
Disallow: /search
Disallow: /Search
Disallow: /settings
Disallow: /shenghuo
Allow: /shop$
Allow: /shop/
Disallow: /shop/collections
Disallow: /shop/productpage
Disallow: /shop/topics
Disallow: /shop?*
Disallow: /social/search?
Disallow: /spbasic
Disallow: /spresults
Disallow: /static/
Disallow: /th$
Disallow: /th?
Disallow: /translator/?ref=
Disallow: /travel/api
Disallow: /travel/css
Disallow: /travel/flight/flightSearch
Disallow: /travel/flight/flightSearchAction
Disallow: /travel/flight/search/?
Disallow: /travel/flight/search?
Disallow: /travel/flights
Disallow: /travel/flight-search
Disallow: /travel/hotel/hotelMiniSearchRequest
Disallow: /travel/hotel/hotelSearch
Disallow: /travel/hotels/checkout
Disallow: /travel/hotels/detail
Disallow: /travel/hotels/search/?
Disallow: /travel/hotels/search?
Disallow: /travel/hotel-search
Disallow: /travel/scripts
Disallow: /travel/secure
Disallow: /url
Disallow: /videos/browsing
Disallow: /videos/explore
Disallow: /videos/favorites
Disallow: /videos/feed
Disallow: /videos/music
Disallow: /videos/trending
Disallow: /widget/cr
Disallow: /widget/entity/search/?
Disallow: /widget/render
Disallow: /widget/snapshot
Disallow: /work

Sitemap: https://www.bing.com/sitemap.xml

Change history

  1. initial snapshot
    • First snapshot of robots.txt archived