NeuralCrawl

The Atlantic / robots.txt snapshot

← back to theatlantic.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 3649 bytes · sha256 4ec505c89c42a966 · raw

final URL: https://www.theatlantic.com/robots.txt

1# As a condition of accessing this website, you agree to abide by the
2# following content signals:
3
4# (a) If a content-signal = yes, you may collect content for the
5# corresponding use.
6# (b) If a content-signal = no, you may not collect content for the
7# corresponding use.
8# (c) If the website operator does not include a content signal for a
9# corresponding use, the website operator neither grants nor restricts
10# permission via content signal with respect to the corresponding use.
11
12# The content signals and their meanings are:
13
14# search: building a search index and providing search results (e.g., returning
15# hyperlinks and short excerpts from your website's contents). Search
16# does not include providing AI-generated search summaries.
17# ai-input: inputting content into one or more AI models (e.g., retrieval
18# augmented generation, grounding, or other real-time taking of
19# content for generative AI search answers).
20# ai-train: training or fine-tuning AI models.
21
22# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
23# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
24# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
25
26User-agent: *
27Disallow: /4624/TheAtlanticOnline/*
28Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/
29Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/*
30Disallow: /ab/*
31Disallow: /video/embed/
32Disallow: /zephr/*
33Disallow: /video/iframe/*
34Disallow: /search/?*q=*
35Allow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/$
36Disallow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/*
37Crawl-delay: 1
38Allow: /
39
40User-agent: AmazonAdBot
41Allow: /
42
43User-agent: Amazonbot
44Disallow: /
45Allow: /feed/*
46
47User-agent: anthropic-ai
48Disallow: /
49
50User-agent: Applebot
51Disallow: /
52
53User-agent: Applebot-Extended
54Disallow: /
55
56User-agent: ArcSpanScraper
57Allow: /
58
59User-agent: AwarioRssBot
60User-agent: AwarioSmartBot
61Disallow: /
62
63User-agent: BingBot
64Content-signal: search=yes, ai-input=no, ai-train=no
65Allow: /
66
67User-agent: Bytespider
68Disallow: /
69
70User-agent: CCBot
71Disallow: /
72
73User-agent: ChatGPT-User
74Allow: /
75
76User-agent: ClaudeBot
77Disallow: /
78
79User-agent: Claude-SearchBot
80Disallow: /
81
82User-agent: Claude-User
83Disallow: /
84
85User-agent: Claude-Web
86Disallow: /
87
88User-agent: cohere-ai
89Disallow: /
90
91User-agent: DataForSeoBot
92Disallow: /
93
94User-agent: Diffbot
95Disallow: /
96
97User-agent: DuckAssistBot
98Content-signal: search=yes, ai-input=no, ai-train=no
99Allow: /
100
101User-agent: FacebookBot
102Content-signal: search=yes, ai-input=no, ai-train=no
103Allow: /
104
105User-agent: Google-Extended
106Disallow: /
107
108User-agent: Googlebot
109Content-signal: search=yes, ai-input=no, ai-train=no
110Disallow: /zephr/*
111Allow: /
112
113User-agent: GPTBot
114Allow: /
115
116User-agent: magpie-crawler
117Disallow: /
118
119User-agent: Meta-ExternalAgent
120Disallow: /
121
122User-agent: Meta-ExternalFetcher
123Disallow: /
124
125User-agent: MistralAI-User
126Disallow: /
127
128User-agent: NewsNow
129Disallow: /
130
131User-agent: news-please
132Disallow: /
133
134User-agent: omgili
135Disallow: /
136
137User-agent: omgilibot
138Disallow: /
139
140User-agent: OAI-SearchBot
141Allow: /
142
143User-agent: Operator
144Allow: /
145
146User-agent: PetalBot
147Disallow: /
148
149User-agent: PerplexityBot
150Disallow: /
151
152User-agent: Perplexity-User
153Disallow: /
154
155User-agent: ProRataInc
156Allow: /
157
158User-agent: Quora-Bot
159Disallow: /
160
161User-agent: Scrapy
162Disallow: /
163
164User-agent: TerraCotta
165Allow: /
166
167User-agent: TimpiBot
168Disallow: /
169
170User-agent: TurnitinBot
171Disallow: /
172
173User-agent: archive.org_bot
174Disallow: /
175
176Sitemap: https://www.theatlantic.com/sitemap.xml
177Sitemap: https://www.theatlantic.com/sponsored/sitemap.xml