NeuralCrawl

๐Ÿ‡ฉ๐Ÿ‡ฐ DSV

dsv.com · European companies · rank #132 · Logistics · live robots.txt ↗

AI crawler access (latest snapshot, 11h ago)

blocked restricted allowed faded = inherited from the * wildcard group

GPTBot
ChatGPT-User
OAI-SearchBot
ClaudeBot
Claude-User
Claude-SearchBot
anthropic-ai
Claude-Web
CCBot
Google-Extended
Applebot-Extended
PerplexityBot
Perplexity-User
Bytespider
Amazonbot
FacebookBot
meta-externalagent
Meta-WebIndexer
meta-externalfetcher
cohere-ai
AI2Bot
Diffbot
omgili
YouBot
DuckAssistBot
MistralAI-User
PanguBot
Timpibot

Current robots.txt 3221 bytes · sha256 c4b8a04ba1a0 · raw

User-agent: *
Sitemap: https://www.dsv.com/sitemap.xml

Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-cs-cz-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-pt-pt-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-es-es-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-nl-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ca-es-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ro-ro-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-uk-ua-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-fr-fr-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-ae-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-gb-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-fr-ca-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-it-it-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ru-ru-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-za-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-us-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-cn-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-es-mx-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-lt-lt-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-nb-no-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-lv-lv-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-sk-sk-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-pt-br-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-sv-se-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-nl-nl-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-zh-cn-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-en-ca-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-fi-fi-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-de-de-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-da-dk-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-bg-bg-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-et-ee-sitemap_index.xml
Sitemap: https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-pl-pl-sitemap_index.xml

Compare any two snapshots

Change history

  1. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  2. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  3. modified +22 −22 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  4. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  5. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  6. modified +21 −21 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  7. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  8. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  9. modified +16 −16 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  10. modified +29 −29 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  11. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  12. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  13. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  14. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  15. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  16. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  17. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  18. modified +20 −20 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  19. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  20. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  21. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  22. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  23. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  24. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  25. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  26. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  27. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  28. modified +23 −23 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  29. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  30. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  31. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  32. modified +13 −13 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  33. modified +30 −30 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  34. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  35. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  36. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  37. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  38. modified +23 −23 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  39. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  40. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  41. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  42. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  43. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  44. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  45. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  46. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  47. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  48. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  49. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  50. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  51. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  52. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  53. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  54. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  55. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  56. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  57. modified +22 −22 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  58. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  59. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  60. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  61. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  62. modified +22 −22 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  63. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  64. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  65. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  66. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  67. modified +22 −22 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  68. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  69. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  70. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  71. modified +23 −23 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  72. modified +29 −29 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  73. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  74. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  75. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  76. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  77. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  78. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  79. modified +22 −22 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  80. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  81. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  82. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  83. modified +29 −29 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  84. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  85. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  86. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  87. modified +29 −29 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  88. modified +30 −30 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  89. modified +23 −23 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  90. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  91. modified +22 −22 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  92. modified +17 −17 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  93. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  94. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  95. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  96. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  97. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  98. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  99. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  100. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  101. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  102. modified +23 −23 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  103. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  104. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  105. modified +22 −22 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  106. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  107. modified +29 −29 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  108. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  109. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  110. modified +25 −25 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  111. modified +26 −26 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  112. modified +18 −18 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  113. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  114. modified +29 −29 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  115. modified +27 −27 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  116. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  117. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  118. modified +7 −7 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  119. modified +24 −24 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  120. modified +28 −28 view diff → snapshot
    • Formatting/comment-only change (no semantic difference)
  121. created +34 −32 view diff → snapshot
    • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-bg-bg-sitemap_index.xml
    • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-ca-es-sitemap_index.xml
    • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-cs-cz-sitemap_index.xml
    • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-da-dk-sitemap_index.xml
    • Added sitemap https://www.dsv.com/service-sitemap-migrated_dsv_dsv-migration-de-de-sitemap_index.xml
    • … 46 more
  122. removed +0 −34
    • robots.txt is no longer served (HTTP 404)
  123. initial snapshot
    • First snapshot of robots.txt archived