🇬🇧 Legal & General

legalandgeneral.com · European companies · rank #85 · Insurance · live robots.txt ↗

AI crawler access (latest snapshot, 13h ago)

⛔blocked restricted ✅allowed faded = inherited from the * wildcard group

✅GPTBot

✅ChatGPT-User

✅OAI-SearchBot

✅ClaudeBot

✅Claude-User

✅Claude-SearchBot

✅anthropic-ai

✅Claude-Web

✅CCBot

✅Google-Extended

✅Applebot-Extended

✅PerplexityBot

✅Perplexity-User

✅Bytespider

✅Amazonbot

✅FacebookBot

✅meta-externalagent

✅meta-externalfetcher

✅cohere-ai

✅AI2Bot

✅Diffbot

✅omgili

✅YouBot

✅DuckAssistBot

✅MistralAI-User

✅PanguBot

✅Timpibot

Current robots.txt 2808 bytes · sha256 3c20fe3540ca · raw

User-agent: *
Disallow: /test_only/
Disallow: /mortgageclub/ar/*
Disallow: /*TeamInfo.xml$
Disallow: /retirement/retirement-options/case-studies/segmenta/*
Disallow: /retirement/retirement-options/case-studies/segmentb/*
Disallow: /retirement/retirement-options/case-studies/segmentc/*
Disallow: /retirement/retirement-options/case-studies/segmentd/*
Disallow: /retirement/retirement-options/case-studies/segmente/*
Disallow: /retirement/retirement-options/case-studies/segmentf/*
Disallow: /retirement/retirement-options/case-studies/segmentg/*
Disallow: /retirement/retirement-options/case-studies/segmenth/*
Disallow: /retirement/retirement-options/case-studies/segmenti/*
Disallow: /retirement/retirement-options/case-studies/segmentj/*
Disallow: /retirement/retirement-options/case-studies/segmentk/*
Disallow: /retirement/retirement-options/case-studies/segmentl/*
Disallow: /retirement/retirement-options/case-studies/segmentm/*
Disallow: /retirement/retirement-options/case-studies/segmentn/*
Disallow: /retirement/retirement-options/case-studies/segmento/*
Disallow: /adviser/mortgage-club/news-and-insight/digest/articles/*
Disallow: /adviser/mortgage-club/sjp/*
Disallow: /co-operative-bank/*
Disallow: /files/library/protection/miscellaneous/q47218lloyds.pdf
Disallow: /leeds/*
Disallow: /landg-assets/adviser/retirement/literature-and-forms/customer-guide/DA400.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/customer-guide/DA401.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/customer-guide/DA403.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/customer-guide/DA404.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/product-summary/DA631.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/product-summary/DA632.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/product-summary/DA633.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/product-summary/DA634.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/terms-and-conditions/DA438.pdf
Disallow: /landg-assets/adviser/retirement/literature-and-forms/application-form/DA702.pdf
Disallow: /web_resources/digital-estate-v3/estate/resources-list.txt
Disallow: /web_resources/digital-estate-v3/estate/css-and-js-list.txt
Disallow: /web_resources/digital-estate-v3/estate/assets/cms-assets/ingestion/HANDLEBAR_REGEX_DE.txt
Disallow: /web_resources/digital-estate-v3/estate/assets/iframe-tools/README.txt
Disallow: /web_resources/digital-estate-v3/estate/version.txt
Disallow: /web_resources/dotcom-elements/prod/resources-list.txt
Disallow: /tools/*

Sitemap: https://www.legalandgeneral.com/sitemap.xml
Sitemap: https://www.legalandgeneral.com/lg-sitemap.xml

Change history

initial 13h ago snapshot
- First snapshot of robots.txt archived