NeuralCrawl

Reliance Steel & Aluminum / robots.txt

← back to rsac.com · change detected 5d ago (2026-08-18T23:34:24Z) · modified +23 −7

What changed

Diff old (2026-08-05T00:54:52Z) → new (2026-08-18T23:34:24Z)

@@ -5,11 +5,27 @@
55  # ---------------------------
66  # END WPFORMS BLOCK
77 
8- # START YOAST BLOCK
9- # ---------------------------
8+ # Search and discovery crawlers that are not useful to this site.
9+ User-agent: AhrefsBot
10+ User-agent: BLEXBot
11+ User-agent: Bytespider
12+ User-agent: CCBot
13+ User-agent: ClaudeBot
14+ User-agent: Claude-SearchBot
15+ User-agent: DataForSeoBot
16+ User-agent: DotBot
17+ User-agent: GPTBot
18+ User-agent: ImagesiftBot
19+ User-agent: MJ12bot
20+ User-agent: PetalBot
21+ User-agent: SemrushBot
22+ User-agent: SeznamBot
23+ User-agent: meta-externalagent
24+ User-agent: omgili
25+ User-agent: omgilibot
26+ Disallow: /
27+
28+ # Permit normal search engines while keeping administrative pages out of results.
1029  User-agent: *
11- Disallow:
12-
13- Sitemap: https://www.reliance.com/sitemap_index.xml
14- # ---------------------------
15- # END YOAST BLOCK
30+ Disallow: /wp-admin/
31+ Allow: /wp-admin/admin-ajax.php