NeuralCrawl

The Atlantic / robots.txt

← back to theatlantic.com · change detected 1d ago (2026-11-05T19:53:48Z) · modified +0 −14

What changed

Diff old (2026-10-28T23:10:43Z) → new (2026-11-05T19:53:48Z)

@@ -24,7 +24,6 @@
2424  # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET.
2525 
2626  User-agent: *
27- Disallow: /4624/TheAtlanticOnline/*
2827  Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/
2928  Disallow: /magazine/archive/2010/11/letters-to-the-editor/308258/*
3029  Disallow: /ab/*
@@ -38,7 +37,6 @@
3837  Allow: /
3938 
4039  User-agent: AmazonAdBot
41- Disallow: /4624/TheAtlanticOnline/*
4240  Disallow: /ab/*
4341  Disallow: /video/embed/
4442  Disallow: /zephr/*
@@ -55,7 +53,6 @@
5553  Disallow: /
5654 
5755  User-agent: Applebot
58- Disallow: /4624/TheAtlanticOnline/*
5956  Disallow: /ab/*
6057  Disallow: /video/embed/
6158  Disallow: /zephr/*
@@ -68,7 +65,6 @@
6865  Disallow: /
6966 
7067  User-agent: ArcSpanScraper
71- Disallow: /4624/TheAtlanticOnline/*
7268  Disallow: /ab/*
7369  Disallow: /video/embed/
7470  Disallow: /zephr/*
@@ -83,7 +79,6 @@
8379 
8480  User-agent: BingBot
8581  Content-signal: search=yes, ai-input=no, ai-train=no
86- Disallow: /4624/TheAtlanticOnline/*
8782  Disallow: /ab/*
8883  Disallow: /video/embed/
8984  Disallow: /zephr/*
@@ -99,7 +94,6 @@
9994  Disallow: /
10095 
10196  User-agent: ChatGPT-User
102- Disallow: /4624/TheAtlanticOnline/*
10397  Disallow: /ab/*
10498  Disallow: /video/embed/
10599  Disallow: /zephr/*
@@ -133,7 +127,6 @@
133127 
134128  User-agent: DuckAssistBot
135129  Content-signal: search=yes, ai-input=no, ai-train=no
136- Disallow: /4624/TheAtlanticOnline/*
137130  Disallow: /ab/*
138131  Disallow: /video/embed/
139132  Disallow: /zephr/*
@@ -144,7 +137,6 @@
144137 
145138  User-agent: FacebookBot
146139  Content-signal: search=yes, ai-input=no, ai-train=no
147- Disallow: /4624/TheAtlanticOnline/*
148140  Disallow: /ab/*
149141  Disallow: /video/embed/
150142  Disallow: /zephr/*
@@ -158,7 +150,6 @@
158150 
159151  User-agent: Googlebot
160152  Content-signal: search=yes, ai-input=no, ai-train=no
161- Disallow: /4624/TheAtlanticOnline/*
162153  Disallow: /ab/*
163154  Disallow: /video/embed/
164155  Disallow: /zephr/*
@@ -168,7 +159,6 @@
168159  Allow: /
169160 
170161  User-agent: GPTBot
171- Disallow: /4624/TheAtlanticOnline/*
172162  Disallow: /ab/*
173163  Disallow: /video/embed/
174164  Disallow: /zephr/*
@@ -202,7 +192,6 @@
202192  Disallow: /
203193 
204194  User-agent: OAI-SearchBot
205- Disallow: /4624/TheAtlanticOnline/*
206195  Disallow: /ab/*
207196  Disallow: /video/embed/
208197  Disallow: /zephr/*
@@ -212,7 +201,6 @@
212201  Allow: /
213202 
214203  User-agent: Operator
215- Disallow: /4624/TheAtlanticOnline/*
216204  Disallow: /ab/*
217205  Disallow: /video/embed/
218206  Disallow: /zephr/*
@@ -233,7 +221,6 @@
233221  Disallow: /
234222 
235223  User-agent: ProRataInc
236- Disallow: /4624/TheAtlanticOnline/*
237224  Disallow: /ab/*
238225  Disallow: /video/embed/
239226  Disallow: /zephr/*
@@ -249,7 +236,6 @@
249236  Disallow: /
250237 
251238  User-agent: TerraCotta
252- Disallow: /4624/TheAtlanticOnline/*
253239  Disallow: /ab/*
254240  Disallow: /video/embed/
255241  Disallow: /zephr/*