Arthur J. Gallagher / robots.txt snapshot
← back to ajg.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 638 bytes · sha256 37294501e7f37147 · raw
final URL: https://www.ajg.com/robots.txt
| 1 | User-agent: Swiftbot |
| 2 | Sitemap: https://www.ajg.com/uk/sitemap.xml |
| 3 | Sitemap: https://www.ajg.com/sitemap.xml |
| 4 | Allow: /uk* |
| 5 | Allow: /* |
| 6 | |
| 7 | User-agent: Elastic-Crawler (8.17.0) Elastic Cloud (https://www.elastic.co/guide/en/cloud/current/ec-get-help.html; 8a66484b73a946289b1008780b9add3c) |
| 8 | Sitemap: https://www.ajg.com/uk/sitemap.xml |
| 9 | Sitemap: https://www.ajg.com/sitemap.xml |
| 10 | Allow: /uk* |
| 11 | Allow: /* |
| 12 | |
| 13 | User-agent: * |
| 14 | Disallow: /us/-/media/files/us/legacy/* |
| 15 | Disallow: /Sitecore |
| 16 | Disallow: /sitecore_files/ |
| 17 | Disallow: /sitecore modules/ |
| 18 | Disallow: /App_Browsers/ |
| 19 | Disallow: /App_config/ |
| 20 | Disallow: /App_Data/ |
| 21 | Disallow: /temp/ |
| 22 | Disallow: /upload/ |
| 23 | Disallow: /xsl/ |