NeuralCrawl

NPR / robots.txt snapshot

← back to npr.org · fetched 2026-09-22T16:26:18Z (1mo ago) · HTTP 200 · 4736 bytes · sha256 53f9d15a3e089e2d · raw

final URL: https://www.npr.org/robots.txt

1# robots.txt for www.npr.org
2# Changes are tracked in www-render
3
4User-agent: *
5Disallow: /mpx/
6Disallow: /cgi-bin
7Disallow: /ramfiles/
8Disallow: /oauth2/
9Disallow: /account/
10Disallow: /proxy/
11Disallow: /*.smil
12Disallow: /*.asx
13Disallow: /*.ram
14Disallow: /*.wav
15Disallow: /*.rmm
16Disallow: /*.js
17Disallow: /*.au
18Disallow: /stations/force/force_localization.php?
19Disallow: /rundowns/segment.php?
20Disallow: /templates/search/*
21Disallow: /2013/03/21/174840895/
22Disallow: /sections/ombudsman/2008/01/frequently_asked_questions_1.html
23Disallow: /sections/health-shots/2013/03/11/173816690/new-voices-for-the-voiceless-synthetic-speech-gets-an-upgrade
24Disallow: /transcripts/470280334*
25Disallow: /2015/07/04/419570939/chasing-memories-in-their-refugee-camp-40-years-after-they-fled-vietnam
26Disallow: /transcripts/419570939*
27Disallow: /sections/parallels/2016/08/15/480128005/for-french-teens-smoking-still-has-more-allure-than-stigma
28Disallow: /transcripts/480128005*
29Disallow: /2020/04/08/830237502/episode-989-what-if-no-one-pays-rent
30Disallow: /transcripts/830237502*
31Disallow: /sections/goatsandsoda/2015/06/09/406744975/a-gender-revolution-hits-the-streets-two-wheels-at-a-time
32Disallow: /transcripts/406744975*
33Disallow: /sureroute
34Disallow: /*/partials*
35Disallow: /*?*
36Disallow: /proxy/*
37Disallow: /player/*
38Disallow: /get/*
39Disallow: /geolocation
40
41# Disallowing the OpenAI web crawler
42User-agent: GPTBot
43Disallow: /
44
45# Disallowing OpenAI plugins
46User-agent: ChatGPT-User
47Disallow: /
48
49# Disallowing Common Crawl
50User-agent: CCBot
51Disallow: /
52
53User-agent: OAI-SearchBot
54Disallow: /
55
56# Disallowing Google Bard and Vertex AI web crawlers
57User-agent: Google-Extended
58Disallow: /
59
60# Disallowing various bots
61User-agent: anthropic-ai
62Disallow: /
63
64User-agent: Applebot-Extended
65Disallow: /
66
67User-agent: Bytespider
68Disallow: /
69
70User-agent: ClaudeBot
71Disallow: /
72
73User-agent: Claude-SearchBot
74Disallow: /
75
76User-agent: Claude-User
77Disallow: /
78
79User-agent: Claude-Web
80Disallow: /
81
82User-agent: cohere-ai
83Disallow: /
84
85User-agent: Diffbot
86Disallow: /
87
88User-agent: FacebookBot
89Disallow: /
90
91User-agent: omgili
92Disallow: /
93
94User-agent: omgilibot
95Disallow: /
96
97User-agent: PerplexityBot
98Disallow: /
99
100User-agent: PerplexityUser
101Disallow: /
102
103# Allow Google Search Console for sitemap crawling
104User-agent: Google-InspectionTool
105Allow: /
106User-agent: Google-Image
107Allow: /
108User-agent: Google-Video
109Allow: /
110User-agent: Googlebot
111Allow: /
112
113# Allowing the TTD contextual crawler for Sponsorship
114User-agent: TTD-Content
115Allow: /
116
117# Allowing the TTD ads fraud/IVT prevention crawler for Sponsorship
118User-agent: Trade Desk ads.txt & sellers.json crawler
119Allow: /
120
121# Allowing the AdsBot-Google crawler for Sponsorship
122User-agent: AdsBot-Google
123Allow: /
124
125# Allowing the AdsBot-Google-Mobile crawler for Sponsorship
126User-agent: AdsBot-Google-Mobile
127Allow: /
128
129# Allowing the Google-Mediapartners crawler for Sponsorship
130User-agent: Mediapartners-Google
131Allow: /
132
133# Allowing the Google-Display-Ads-Bot crawler for Sponsorship
134User-agent: Google-Display-Ads-Bot
135Allow: /
136
137# Allowing the IAB tech lab crawler for Sponsorship
138User-agent: IAB-Tech-Lab
139Allow: /
140
141# Allowing the IAS crawler for Sponsorship
142User-agent: ias_crawler
143Allow: /
144
145# Allowing the IAS Wombles crawler for Sponsorship
146User-agent: ias_wombles
147Allow: /
148
149# Allowing the Amazon Standards crawler for Sponsorship
150User-agent: Amazon-Advertising-ad-standards-bot/1.0
151Allow: /
152# (fetches ads.txt, app-ads.txt, sellers.json)
153User-agent: APS-ad-standards-bot/1.0
154Allow: /
155
156# New requirement as of 3/2/26 to be eligible for demand from Amazon ads.
157User-agent: AmazonAdBot
158Allow: /
159
160# Allowing the Audigent crawler for Sponsorship
161User-agent: AudigentAdBot
162Allow: /
163
164# Allowing the Concert crawler for Sponsorship
165User-agent: Concert/1.0
166Allow: /
167
168# Allowing Centro/Basis
169User-agent: Centro
170Allow: /
171
172# Allowing BidSwitch (AdX DSP)
173User-agent: bidswitchbot/1.0
174Allow: /
175
176# Allowing Roku (AdX DSP)
177User-agent: DataXu/1.0
178Allow: /
179
180# Allowing PubMatic
181User-agent: PubMatic
182Allow: /
183
184# Allowing AdForm
185User-agent: IAB ATQ team adform robot site.adform.com/general/url-fetcher
186Allow: /
187
188# Allowing StackAdapt
189User-agent: StackAdapt
190Allow: /
191
192# Allowing Quantcast
193User-agent: Quantcastbot/1.0
194Allow: /
195User-agent: Quantcastbot/2.0
196Allow: /
197
198# Facebook sharing
199User-agent: facebookexternalhit
200Allow: /
201
202# Ensures that we're using the correct sitemap. The fact that this is googlecrawl*.npr.org is OK because the crawler will only accept
203# URLs in this sitemap to match www*.npr.org
204Sitemap: https://googlecrawl.npr.org/standard/sitemap_index.xml
205Sitemap: https://googlecrawl.npr.org/news/sitemap_news.xml
206Sitemap: https://googlecrawl.npr.org/video/sitemap_index.xml
207Sitemap: https://www.npr.org/live-updates/sitemap.xml