NeuralCrawl

NPR / robots.txt snapshot

← back to npr.org · fetched 2026-10-26T22:30:46Z (11d ago) · HTTP 200 · 4780 bytes · sha256 6564584f1b7342ee · raw

final URL: https://www.npr.org/robots.txt

1# robots.txt for www.npr.org
2# Changes are tracked in www-render
3
4User-agent: *
5Disallow: /mpx/
6Disallow: /cgi-bin
7Disallow: /ramfiles/
8Disallow: /oauth2/
9Disallow: /account/
10Disallow: /proxy/
11Disallow: /*.smil
12Disallow: /*.asx
13Disallow: /*.ram
14Disallow: /*.wav
15Disallow: /*.rmm
16Disallow: /*.js
17Disallow: /*.au
18Disallow: /stations/force/force_localization.php?
19Disallow: /rundowns/segment.php?
20Disallow: /templates/search/*
21Disallow: /2013/03/21/174840895/
22Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html
23Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade
24Disallow: /transcripts/470280334*
25Disallow: /2015/07/04/419570939/chasing-memories-in-their-refugee-camp-40-years-after-they-fled-vietnam
26Disallow: /transcripts/419570939*
27Disallow: /sections/parallels/2016/08/15/480128005/for-french-teens-smoking-still-has-more-allure-than-stigma
28Disallow: /transcripts/480128005*
29Disallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rent
30Disallow: /transcripts/830237502*
31Disallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-time
32Disallow: /transcripts/406744975*
33Disallow: /sureroute
34Disallow: /*/partials*
35Disallow: /*?*
36Disallow: /proxy/*
37Disallow: /player/*
38Disallow: /get/*
39Disallow: /geolocation
40
41# Disallowing the OpenAI web crawler
42User-agent: GPTBot
43Disallow: /
44
45# Disallowing OpenAI plugins
46User-agent: ChatGPT-User
47Disallow: /
48
49# Disallowing Common Crawl
50User-agent: CCBot
51Disallow: /
52
53User-agent: OAI-SearchBot
54Disallow: /
55
56# Disallowing Google Bard and Vertex AI web crawlers
57User-agent: Google-Extended
58Disallow: /
59
60# Disallowing various bots
61User-agent: anthropic-ai
62Disallow: /
63
64User-agent: Applebot-Extended
65Disallow: /
66
67User-agent: Bytespider
68Disallow: /
69
70User-agent: ClaudeBot
71Disallow: /
72
73User-agent: Claude-SearchBot
74Disallow: /
75
76User-agent: Claude-User
77Disallow: /
78
79User-agent: Claude-Web
80Disallow: /
81
82User-agent: cohere-ai
83Disallow: /
84
85User-agent: Diffbot
86Disallow: /
87
88User-agent: FacebookBot
89Disallow: /
90
91User-agent: Meta-ExternalAgent
92Disallow: /
93
94User-agent: omgili
95Disallow: /
96
97User-agent: omgilibot
98Disallow: /
99
100User-agent: PerplexityBot
101Disallow: /
102
103User-agent: PerplexityUser
104Disallow: /
105
106# Allow Google Search Console for sitemap crawling
107User-agent: Google-InspectionTool
108Allow: /
109User-agent: Google-Image
110Allow: /
111User-agent: Google-Video
112Allow: /
113User-agent: Googlebot
114Allow: /
115
116# Allowing the TTD contextual crawler for Sponsorship
117User-agent: TTD-Content
118Allow: /
119
120# Allowing the TTD ads fraud/IVT prevention crawler for Sponsorship
121User-agent: Trade Desk ads.txt & sellers.json crawler
122Allow: /
123
124# Allowing the AdsBot-Google crawler for Sponsorship
125User-agent: AdsBot-Google
126Allow: /
127
128# Allowing the AdsBot-Google-Mobile crawler for Sponsorship
129User-agent: AdsBot-Google-Mobile
130Allow: /
131
132# Allowing the Google-Mediapartners crawler for Sponsorship
133User-agent: Mediapartners-Google
134Allow: /
135
136# Allowing the Google-Display-Ads-Bot crawler for Sponsorship
137User-agent: Google-Display-Ads-Bot
138Allow: /
139
140# Allowing the IAB tech lab crawler for Sponsorship
141User-agent: IAB-Tech-Lab
142Allow: /
143
144# Allowing the IAS crawler for Sponsorship
145User-agent: ias_crawler
146Allow: /
147
148# Allowing the IAS Wombles crawler for Sponsorship
149User-agent: ias_wombles
150Allow: /
151
152# Allowing the Amazon Standards crawler for Sponsorship
153User-agent: Amazon-Advertising-ad-standards-bot/1.0
154Allow: /
155# (fetches ads.txt, app-ads.txt, sellers.json)
156User-agent: APS-ad-standards-bot/1.0
157Allow: /
158
159# New requirement as of 3/2/26 to be eligible for demand from Amazon ads.
160User-agent: AmazonAdBot
161Allow: /
162
163# Allowing the Audigent crawler for Sponsorship
164User-agent: AudigentAdBot
165Allow: /
166
167# Allowing the Concert crawler for Sponsorship
168User-agent: Concert/1.0
169Allow: /
170
171# Allowing Centro/Basis
172User-agent: Centro
173Allow: /
174
175# Allowing BidSwitch (AdX DSP)
176User-agent: bidswitchbot/1.0
177Allow: /
178
179# Allowing Roku (AdX DSP)
180User-agent: DataXu/1.0
181Allow: /
182
183# Allowing PubMatic
184User-agent: PubMatic
185Allow: /
186
187# Allowing AdForm
188User-agent: IAB ATQ team adform robot site.adform.com/general/url-fetcher
189Allow: /
190
191# Allowing StackAdapt
192User-agent: StackAdapt
193Allow: /
194
195# Allowing Quantcast
196User-agent: Quantcastbot/1.0
197Allow: /
198User-agent: Quantcastbot/2.0
199Allow: /
200
201# Facebook sharing
202User-agent: facebookexternalhit
203Allow: /
204
205# Ensures that we're using the correct sitemap. The fact that this is googlecrawl*.npr.org is OK because the crawler will only accept
206# URLs in this sitemap to match www*.npr.org
207Sitemap: https://googlecrawl.npr.org/standard/sitemap_index.xml
208Sitemap: https://googlecrawl.npr.org/news/sitemap_news.xml
209Sitemap: https://googlecrawl.npr.org/video/sitemap_index.xml
210Sitemap: https://www.npr.org/live-updates/sitemap.xml