NeuralCrawl

The Associated Press / robots.txt snapshot

← back to apnews.com · fetched 2026-10-15T18:35:45Z (22d ago) · HTTP 200 · 1349 bytes · sha256 d4f9a68d011b57af · raw

final URL: https://apnews.com/robots.txt

1User-Agent: *
2Disallow: *_ptid=*
3Disallow: *?prx_t=*
4Disallow: /zephr*
5Disallow: /blaize*
6Disallow: /search?q=*
7Disallow: /*.rss
8Disallow: /buyline-shopping/*
9Disallow: /buyline-personal-finance/*
10Disallow: *?jw_start
11Disallow: *&jw_start
12Disallow: /gallery/
13Disallow: /config/newrelic/prod
14Disallow: /parsely-api-endpoint
15Disallow: /apdata/
16Disallow: /article/border-immigration-election-2182fec11fa008ec96c4202e5656d19e
17Disallow: /api/v2/feed/
18Disallow: /press-release/*
19Disallow: /*/widgets/
20Disallow: /*/widget/
21Disallow: /*/scripts/
22
23Sitemap: https://apnews.com/ap-sitemap.xml
24Sitemap: https://apnews.com/news-sitemap-content.xml
25Sitemap: https://apnews.com/hubs-sitemap-content.xml
26Sitemap: https://apnews.com/video-sitemap.xml
27Sitemap: https://apnews.com/author-sitemap-content.xml
28Sitemap: https://apnews.com/projects/elections-2026/sitemap-index.xml
29
30# Disallow Rules
31
32User-agent: CCBot
33Disallow: /
34
35User-agent: GPTBot
36Disallow: /
37
38User-agent: anthropic-ai
39Disallow: /
40
41User-agent: ClaudeBot
42Disallow: /
43
44User-agent: Claude-Web
45Disallow: /
46
47User-agent: cohere-ai
48Disallow: /
49
50User-agent: PerplexityBot
51Disallow: /
52
53User-agent: Amazonbot
54Disallow: /
55
56User-agent: Applebot-Extended
57Disallow: /
58
59User-agent: Bytedance
60Disallow: /
61
62User-agent: Timpibot
63Disallow: /
64
65User-agent: Claude-User
66Disallow: /
67
68User-agent: Claude-SearchBot
69Disallow: /