NeuralCrawl

The Associated Press / robots.txt snapshot

← back to apnews.com · fetched 2026-08-22T00:26:44Z (2mo ago) · HTTP 200 · 1294 bytes · sha256 c04780bfd1f42e75 · raw

final URL: https://apnews.com/robots.txt

1User-Agent: *
2Disallow:
3Disallow: *_ptid=*
4Disallow: *?prx_t=*
5Disallow: /zephr*
6Disallow: /blaize*
7Disallow: /search?q=*
8Disallow: /*.rss
9Disallow: /buyline-shopping/*
10Disallow: /buyline-personal-finance/*
11Disallow: *?jw_start
12Disallow: *&jw_start
13Disallow: /gallery/
14Disallow: /config/newrelic/prod
15Disallow: /parsely-api-endpoint
16Disallow: /apdata/
17Disallow: /article/border-immigration-election-2182fec11fa008ec96c4202e5656d19e
18Disallow: /api/v2/feed/
19Disallow: /press-release/*
20
21Sitemap: https://apnews.com/ap-sitemap.xml
22Sitemap: https://apnews.com/news-sitemap-content.xml
23Sitemap: https://apnews.com/hubs-sitemap-content.xml
24Sitemap: https://apnews.com/video-sitemap.xml
25Sitemap: https://apnews.com/author-sitemap-content.xml
26Sitemap: https://apnews.com/projects/elections-2026/sitemap-index.xml
27
28# Disallow Rules
29
30User-agent: CCBot
31Disallow: /
32
33User-agent: GPTBot
34Disallow: /
35
36User-agent: anthropic-ai
37Disallow: /
38
39User-agent: ClaudeBot
40Disallow: /
41
42User-agent: Claude-Web
43Disallow: /
44
45User-agent: cohere-ai
46Disallow: /
47
48User-agent: PerplexityBot
49Disallow: /
50
51User-agent: Amazonbot
52Disallow: /
53
54User-agent: Applebot-Extended
55Disallow: /
56
57User-agent: Bytedance
58Disallow: /
59
60User-agent: Timpibot
61Disallow: /
62
63User-agent: Claude-User
64Disallow: /
65
66User-agent: Claude-SearchBot
67Disallow: /