NeuralCrawl

Penguin Random House / robots.txt

← back to penguinrandomhouse.com · change detected 1mo ago (2026-09-21T16:12:30Z) · modified +11 −1

What changed

Diff old (2026-09-17T15:11:14Z) → new (2026-09-21T16:12:30Z)

@@ -1,2 +1,12 @@
11  User-agent: *
2- Disallow: /
2+ Disallow: /wp-admin/
3+ Disallow: /wp-includes/
4+ Disallow: /wp-content/plugins/
5+ Disallow: /wp-content/cache/
6+ Disallow: /trackback/
7+ Disallow: /search/
8+ Disallow: /prh-internal-news/
9+ Disallow: /interactive/reading-preference
10+ Allow: /wp-admin/admin-ajax.php
11+ Allow: /wp-includes/css/dist/block-library/style.min.css
12+