🇪🇸 Repsol
repsol.com · European companies · rank #56 · Energy · live robots.txt ↗
AI crawler access (latest snapshot, 13h ago)
⛔blocked
restricted
✅allowed
faded = inherited from the * wildcard group
✅GPTBot
✅ChatGPT-User
✅OAI-SearchBot
✅ClaudeBot
✅Claude-User
✅Claude-SearchBot
✅anthropic-ai
✅Claude-Web
✅CCBot
✅Google-Extended
✅Applebot-Extended
✅PerplexityBot
✅Perplexity-User
✅Bytespider
✅Amazonbot
✅FacebookBot
✅meta-externalagent
✅meta-externalfetcher
✅cohere-ai
✅AI2Bot
✅Diffbot
✅omgili
✅YouBot
✅DuckAssistBot
✅MistralAI-User
✅PanguBot
✅Timpibot
Current robots.txt 1352 bytes · sha256 d7401ea1c38e · raw
User-agent: * Disallow: /*/dynamic/* Disallow: /es/energia-innovacion/un-mundo-colaborativo/energy-ventures/index.cshtml Disallow: /en/energy-and-innovation/a-collaborative-world/energy-ventures/index.cshtml # Archivos Disallow: *.xls Disallow: *.zip Disallow: *.mp4 Disallow: *.m4v # Meses Disallow: *month Allow: *month=all # Directorios Antiguos Herramientas Disallow: */aplicaciones/SA/Herramientas/* Disallow: */SA/Corporacion/BuscadorCertificadosCorporacion/* Disallow: */SA/Corporacion/Newsletter/* Disallow: */SA/Herramientas/FormsLogin/* Disallow: */SA/Herramientas/NuevoRegistroParticulares/* Disallow: */herramientas/registro/* Disallow: */fenix/FormularioProveedorRemApp/* Disallow: */includes/_t # Directorios Antiguos Disallow: */prensa/news/* Disallow: /SE/Motor/ Disallow: /test/ Disallow: /demo_no_eliminar/ Disallow: /SE/Competicion/ Disallow: *?gp* Disallow: */EESS_Portugal/* Disallow: */EESS_Peru/* Disallow: */SolredMasPTUNIVERSIA/* Disallow: */Solred/SolredMasRACC/* Disallow: *aspx?* Allow: /es/productos-y-servicios/estaciones-de-servicio/productos/carburantes-neotech/index.cshtml # Otros Disallow: */coronavirus/ Sitemap:https://www.repsol.com/es_paginas_sitemap.xml Sitemap:https://www.repsol.com/es_prensa_sitemap.xml Sitemap:https://www.repsol.com/en_pages_sitemap.xml Sitemap:https://www.repsol.com/en_press_sitemap.xml
Change history
-
initial snapshot
- First snapshot of robots.txt archived