NeuralCrawl

🇺🇸 MetLife

metlife.com · US 500 · rank #60 · Financials · live robots.txt ↗

AI crawler access (latest snapshot, 3d ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅Meta-WebIndexer
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot

Current robots.txt 1985 bytes · sha256 2a82db4d667f · raw 

User-agent: *
Disallow: /sitemap/_jcr_content/root/sitemap.xml
Disallow: /*aemform.iframe.html*
Disallow: /uat/
Disallow: /smoketest/
Disallow: /pagenotfound/
Disallow: /*.pdf
Disallow: /*.pdf$
Disallow: https://origin-intl.metlife.com/*
Disallow: https://target.www.metlife.com/*
Disallow: https://origin-target.metlife.com/*
User-agent: prodGSA
Disallow: /system/
Disallow: /MCOMDefault/
Disallow: /WPSAssets/
Disallow: /navimages/
Disallow: /*.css$
Disallow: /*.js$
Disallow: /pagenotfound/
User-agent: drGSA
Disallow: /system/
Disallow: /MCOMDefault/
Disallow: /WPSAssets/
Disallow: /navimages/
Disallow: /*.css$
Disallow: /*.js$
Disallow: /pagenotfound/
User-agent: devqaGSA
Disallow: /system/
Disallow: /MCOMDefault/
Disallow: /WPSAssets/
Disallow: /navimages/
Disallow: /*.css$
Disallow: /*.js$
Sitemap: https://www.metlife.com/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-us/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-au/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-ca/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-ch/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-de/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-dk/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-es/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-fi/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-uk/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-ie/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-nl/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-se/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-sg/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-hk/sitemap.xml
Sitemap: https://www.metlife.com/investments/zh-hk/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-me/sitemap.xml
Sitemap: https://www.metlife.com/investments/global/sitemap.xml
Sitemap: https://www.metlife.com/investments/en-latam/sitemap.xml

Compare any two snapshots

Change history

  1. modified +1 −0 view diff → snapshot
    • Added sitemap https://www.metlife.com/investments/global/sitemap.xml
  2. modified +0 −1 view diff → snapshot
    • Removed sitemap https://www.metlife.com/investments/global/sitemap.xml
  3. modified +1 −0 view diff → snapshot
    • Added sitemap https://www.metlife.com/investments/global/sitemap.xml
  4. modified +0 −1 view diff → snapshot
    • Removed sitemap https://www.metlife.com/investments/global/sitemap.xml
  5. modified +1 −0 view diff → snapshot
    • Added sitemap https://www.metlife.com/investments/global/sitemap.xml
  6. modified +17 −8 view diff → snapshot
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 19 more
  7. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  8. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  9. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  10. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  11. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  12. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  13. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  14. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  15. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  16. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  17. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  18. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  19. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  20. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  21. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  22. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  23. modified +8 −2 view diff → snapshot
    • Added Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • Added Disallow: https://metlife-prod.adobecqms.net/* for *
    • Added Disallow: https://metlife-prodtenants.adobecqms.net/* for *
    • … 4 more
  24. modified +2 −8 view diff → snapshot
    • Added Disallow: /investments$ for *
    • Added Disallow: /investments/ for *
    • Removed Disallow: https://author-metlife-prod.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-2019.adobecqms.net/* for *
    • Removed Disallow: https://metlife-prod-65.adobecqms.net/* for *
    • … 4 more
  25. initial snapshot
    • First snapshot of robots.txt archived