🇺🇸 MongoDB
mongodb.com · Top 1000 websites · rank #883 · Programming and Developer Software · live robots.txt ↗
AI crawler access (latest snapshot, 13h ago)
⛔blocked
restricted
✅allowed
faded = inherited from the * wildcard group
✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot
Current robots.txt 2823 bytes · sha256 817d348c26b8 · raw
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file User-agent: mauibot Disallow: /community/forums/ User-agent: semrushbot Disallow: /community/forums/ User-agent: ahrefsbot Disallow: /community/forums/ User-agent: blexbot Disallow: /community/forums/ User-agent: seo spider Disallow: /community/forums/ User-agent: bingbot Disallow: /community/forums/auth/ Disallow: /community/forums/assets/browser-update*.js Disallow: /community/forums/email/ Disallow: /community/forums/session Disallow: /community/forums/session/ Disallow: /community/forums/admin Disallow: /community/forums/admin/ Disallow: /community/forums/user-api-key Disallow: /community/forums/user-api-key/ Disallow: /community/forums/*?api_key* Disallow: /community/forums/*?*api_key* User-agent: * # Directories Disallow: /misc/ Disallow: /profiles/ Disallow: /u/ Disallow: /community/forums/auth/ Disallow: /community/forums/email/ Disallow: /community/forums/session Disallow: /community/forums/session/ Disallow: /community/forums/admin Disallow: /community/forums/admin/ Disallow: /community/forums/user-api-key Disallow: /community/forums/user-api-key/ Disallow: /community/forums/g/ Disallow: /community/forums/u/ Disallow: /community/forums/user_avatar/ # Files Disallow: /admin/* Disallow: /data/* Disallow: /comment/reply/ Disallow: /filter/tips/ Disallow: /node/add/ Disallow: /search/ Disallow: /search Disallow: /user/register/ Disallow: /user/password/ Disallow: /user/login/ Disallow: /user/logout/ Disallow: /lp/p/* Disallow: /lp/s/* Disallow: /lp/d/* Disallow: /thank-you/* Disallow: /link/* Disallow: /community/forums/assets/browser-update*.js Disallow: /docs/*.epub Disallow: /docs/*.pdf Disallow: /docs/atlas/app-services/ # No access for quicktabs in the URL Disallow: /*?quicktabs_* Disallow: /*&quicktabs_* Disallow: /*?qt-view_* Disallow: /*&qt-view_* # No crawling field collections or taxonomy Disallow: /field-collection/* Disallow: /tags/* Disallow: /taxonomy/term/* # Paths (no clean URLs) Disallow: /?q=admin/ Disallow: /?q=comment/reply/ Disallow: /?q=filter/tips/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ Disallow: /?q=user/logout/ # Specific pages Disallow: /commercialsupport/ Disallow: /partners/referral-federal Disallow: /customer-evaluation-downloads-development-versions Disallow: /community/forums/*?api_key* Disallow: /community/forums/*?*api_key* Disallow: /docs/meta/ Disallow: /docs/search/ Disallow: /docs/openapi/ # Prevent exposing future docs Disallow: /docs-qa* # Prevent exposing devhub UAT Disallow: /devcenter-uat* Disallow: /lp/internal/cheq-ppc-invalid-users* Disallow: /.well-known/security.txt Sitemap: https://www.mongodb.com/sitemap-index.xml
Change history
-
initial snapshot
- First snapshot of robots.txt archived