NeuralCrawl

The Atlantic / robots.txt snapshot

← back to theatlantic.com · fetched 2026-09-17T21:12:52Z (1d ago) · HTTP 200 · 3722 bytes · sha256 500c1ee9d0c39ec3 · raw

final URL: https://www.theatlantic.com/robots.txt

1# As a condition of accessing this website, you agree to abide by the
2# following content signals:
3
4# (a) If a content-signal = yes, you may collect content for the
5# corresponding use.
6# (b) If a content-signal = no, you may not collect content for the
7# corresponding use.
8# (c) If the website operator does not include a content signal for a
9# corresponding use, the website operator neither grants nor restricts
10# permission via content signal with respect to the corresponding use.
11
12# The content signals and their meanings are:
13
14# search: building a search index and providing search results (e.g., returning
15# hyperlinks and short excerpts from your website's contents). Search
16# does not include providing AI-generated search summaries.
17# ai-input: inputting content into one or more AI models (e.g., retrieval
18# augmented generation, grounding, or other real-time taking of
19# content for generative AI search answers).
20# ai-train: training or fine-tuning AI models.
21
22# ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF
23# RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT
24# AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
25
26User-agent: *
27Disallow: /4624/TheAtlanticOnline/*
28Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/
29Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/*
30Disallow: /ab/*
31Disallow: /video/embed/
32Disallow: /zephr/*
33Disallow: /video/iframe/*
34Disallow: /search/?*q=*
35Allow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/$
36Disallow: /magazine/archive/2001/02/bill-clinton-and-his-consequences/303383/*
37Crawl-delay: 1
38Allow: /
39
40User-agent: AmazonAdBot
41Allow: /
42
43User-agent: Amazonbot
44Disallow: /
45Allow: /feed/*
46
47User-agent: anthropic-ai
48Disallow: /
49
50User-agent: Applebot
51Allow: /
52
53User-agent: Applebot-Extended
54Disallow: /
55
56User-agent: ArcSpanScraper
57Allow: /
58
59User-agent: AwarioRssBot
60User-agent: AwarioSmartBot
61Disallow: /
62
63User-agent: BingBot
64Content-signal: search=yes, ai-input=no, ai-train=no
65Allow: /
66
67User-agent: Bytespider
68Disallow: /
69
70User-agent: CCBot
71Disallow: /
72
73User-agent: ChatGPT-User
74Allow: /
75
76User-agent: ClaudeBot
77Disallow: /
78
79User-agent: Claude-SearchBot
80Allow: /sponsored/
81Disallow: /
82
83User-agent: Claude-User
84Allow: /sponsored/
85Disallow: /
86
87User-agent: Claude-Web
88Disallow: /
89
90User-agent: cohere-ai
91Disallow: /
92
93User-agent: DataForSeoBot
94Disallow: /
95
96User-agent: Diffbot
97Disallow: /
98
99User-agent: DuckAssistBot
100Content-signal: search=yes, ai-input=no, ai-train=no
101Allow: /
102
103User-agent: FacebookBot
104Content-signal: search=yes, ai-input=no, ai-train=no
105Allow: /
106
107User-agent: Google-Extended
108Disallow: /
109
110User-agent: Googlebot
111Content-signal: search=yes, ai-input=no, ai-train=no
112Disallow: /zephr/*
113Allow: /
114
115User-agent: GPTBot
116Allow: /
117
118User-agent: magpie-crawler
119Disallow: /
120
121User-agent: Meta-ExternalAgent
122Disallow: /
123
124User-agent: Meta-ExternalFetcher
125Disallow: /
126
127User-agent: MistralAI-User
128Disallow: /
129
130User-agent: NewsNow
131Disallow: /
132
133User-agent: news-please
134Disallow: /
135
136User-agent: omgili
137Disallow: /
138
139User-agent: omgilibot
140Disallow: /
141
142User-agent: OAI-SearchBot
143Allow: /
144
145User-agent: Operator
146Allow: /
147
148User-agent: PetalBot
149Disallow: /
150
151User-agent: PerplexityBot
152Allow: /sponsored/
153Disallow: /
154
155User-agent: Perplexity-User
156Allow: /sponsored/
157Disallow: /
158
159User-agent: ProRataInc
160Allow: /
161
162User-agent: Quora-Bot
163Disallow: /
164
165User-agent: Scrapy
166Disallow: /
167
168User-agent: TerraCotta
169Allow: /
170
171User-agent: TimpiBot
172Disallow: /
173
174User-agent: TurnitinBot
175Disallow: /
176
177User-agent: archive.org_bot
178Disallow: /
179
180Sitemap: https://www.theatlantic.com/sitemap.xml
181Sitemap: https://www.theatlantic.com/sponsored/sitemap.xml