NeuralCrawl

The Atlantic / robots.txt

← back to theatlantic.com · change detected 9d ago (2026-10-28T23:10:43Z) · modified +90 −0

What changed

Diff old (2026-09-17T21:12:52Z) → new (2026-10-28T23:10:43Z)

@@ -38,6 +38,13 @@
3838  Allow: /
3939 
4040  User-agent: AmazonAdBot
41+ Disallow: /4624/TheAtlanticOnline/*
42+ Disallow: /ab/*
43+ Disallow: /video/embed/
44+ Disallow: /zephr/*
45+ Disallow: /video/iframe/*
46+ Disallow: /search/?*q=*
47+ Crawl-delay: 1
4148  Allow: /
4249 
4350  User-agent: Amazonbot
@@ -48,12 +55,26 @@
4855  Disallow: /
4956 
5057  User-agent: Applebot
58+ Disallow: /4624/TheAtlanticOnline/*
59+ Disallow: /ab/*
60+ Disallow: /video/embed/
61+ Disallow: /zephr/*
62+ Disallow: /video/iframe/*
63+ Disallow: /search/?*q=*
64+ Crawl-delay: 1
5165  Allow: /
5266 
5367  User-agent: Applebot-Extended
5468  Disallow: /
5569 
5670  User-agent: ArcSpanScraper
71+ Disallow: /4624/TheAtlanticOnline/*
72+ Disallow: /ab/*
73+ Disallow: /video/embed/
74+ Disallow: /zephr/*
75+ Disallow: /video/iframe/*
76+ Disallow: /search/?*q=*
77+ Crawl-delay: 1
5778  Allow: /
5879 
5980  User-agent: AwarioRssBot
@@ -62,6 +83,13 @@
6283 
6384  User-agent: BingBot
6485  Content-signal: search=yes, ai-input=no, ai-train=no
86+ Disallow: /4624/TheAtlanticOnline/*
87+ Disallow: /ab/*
88+ Disallow: /video/embed/
89+ Disallow: /zephr/*
90+ Disallow: /video/iframe/*
91+ Disallow: /search/?*q=*
92+ Crawl-delay: 1
6593  Allow: /
6694 
6795  User-agent: Bytespider
@@ -71,6 +99,13 @@
7199  Disallow: /
72100 
73101  User-agent: ChatGPT-User
102+ Disallow: /4624/TheAtlanticOnline/*
103+ Disallow: /ab/*
104+ Disallow: /video/embed/
105+ Disallow: /zephr/*
106+ Disallow: /video/iframe/*
107+ Disallow: /search/?*q=*
108+ Crawl-delay: 1
74109  Allow: /
75110 
76111  User-agent: ClaudeBot
@@ -98,10 +133,24 @@
98133 
99134  User-agent: DuckAssistBot
100135  Content-signal: search=yes, ai-input=no, ai-train=no
136+ Disallow: /4624/TheAtlanticOnline/*
137+ Disallow: /ab/*
138+ Disallow: /video/embed/
139+ Disallow: /zephr/*
140+ Disallow: /video/iframe/*
141+ Disallow: /search/?*q=*
142+ Crawl-delay: 1
101143  Allow: /
102144 
103145  User-agent: FacebookBot
104146  Content-signal: search=yes, ai-input=no, ai-train=no
147+ Disallow: /4624/TheAtlanticOnline/*
148+ Disallow: /ab/*
149+ Disallow: /video/embed/
150+ Disallow: /zephr/*
151+ Disallow: /video/iframe/*
152+ Disallow: /search/?*q=*
153+ Crawl-delay: 1
105154  Allow: /
106155 
107156  User-agent: Google-Extended
@@ -109,10 +158,23 @@
109158 
110159  User-agent: Googlebot
111160  Content-signal: search=yes, ai-input=no, ai-train=no
161+ Disallow: /4624/TheAtlanticOnline/*
162+ Disallow: /ab/*
163+ Disallow: /video/embed/
112164  Disallow: /zephr/*
165+ Disallow: /video/iframe/*
166+ Disallow: /search/?*q=*
167+ Crawl-delay: 1
113168  Allow: /
114169 
115170  User-agent: GPTBot
171+ Disallow: /4624/TheAtlanticOnline/*
172+ Disallow: /ab/*
173+ Disallow: /video/embed/
174+ Disallow: /zephr/*
175+ Disallow: /video/iframe/*
176+ Disallow: /search/?*q=*
177+ Crawl-delay: 1
116178  Allow: /
117179 
118180  User-agent: magpie-crawler
@@ -140,9 +202,23 @@
140202  Disallow: /
141203 
142204  User-agent: OAI-SearchBot
205+ Disallow: /4624/TheAtlanticOnline/*
206+ Disallow: /ab/*
207+ Disallow: /video/embed/
208+ Disallow: /zephr/*
209+ Disallow: /video/iframe/*
210+ Disallow: /search/?*q=*
211+ Crawl-delay: 1
143212  Allow: /
144213 
145214  User-agent: Operator
215+ Disallow: /4624/TheAtlanticOnline/*
216+ Disallow: /ab/*
217+ Disallow: /video/embed/
218+ Disallow: /zephr/*
219+ Disallow: /video/iframe/*
220+ Disallow: /search/?*q=*
221+ Crawl-delay: 1
146222  Allow: /
147223 
148224  User-agent: PetalBot
@@ -157,6 +233,13 @@
157233  Disallow: /
158234 
159235  User-agent: ProRataInc
236+ Disallow: /4624/TheAtlanticOnline/*
237+ Disallow: /ab/*
238+ Disallow: /video/embed/
239+ Disallow: /zephr/*
240+ Disallow: /video/iframe/*
241+ Disallow: /search/?*q=*
242+ Crawl-delay: 1
160243  Allow: /
161244 
162245  User-agent: Quora-Bot
@@ -166,6 +249,13 @@
166249  Disallow: /
167250 
168251  User-agent: TerraCotta
252+ Disallow: /4624/TheAtlanticOnline/*
253+ Disallow: /ab/*
254+ Disallow: /video/embed/
255+ Disallow: /zephr/*
256+ Disallow: /video/iframe/*
257+ Disallow: /search/?*q=*
258+ Crawl-delay: 1
169259  Allow: /
170260 
171261  User-agent: TimpiBot