NeuralCrawl

Newsweek / robots.txt snapshot

← back to newsweek.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 3852 bytes · sha256 41a77e14415bfb19 · raw

final URL: https://www.newsweek.com/robots.txt

1User-agent: *
2Disallow: /feeds/*
3Disallow: /system/
4Disallow: /addineyeV2l
5Disallow: /eyeblaster
6Disallow: /doubleclick/
7Disallow: /ads/sponsored
8
9# Disallow Bots
10User-agent: Amazonbot
11Disallow: /
12User-agent: anthropic-ai
13Disallow: /
14User-agent: Applebot-Extended
15Disallow: /
16User-agent: AwarioRssBot
17Disallow: /
18User-agent: AwarioSmartBot
19Disallow: /
20User-agent: Bytespider
21Disallow: /
22User-agent: CCBot
23Disallow: /
24User-agent: ChatGPT-User
25Disallow: /
26User-agent: ClaudeBot
27Disallow: /
28User-agent: Claude-Web
29Disallow: /
30User-agent: cohere-ai
31Disallow: /
32User-agent: DataForSeoBot
33Disallow: /
34User-agent: Diffbot
35Disallow: /
36User-agent: FacebookBot
37Disallow: /
38User-agent: GPTBot
39Disallow: /
40User-agent: magpie-crawler
41Disallow: /
42User-agent: NewsNow
43Disallow: /
44User-agent: news-please
45Disallow: /
46User-agent: omgili
47Disallow: /
48User-agent: omgilibot
49Disallow: /
50User-agent: peer39_crawler
51Disallow: /
52User-agent: peer39_crawler/1.0
53Disallow: /
54User-agent: PerplexityBot
55Disallow: /
56User-agent: Scrapy
57Disallow: /
58User-agent: TurnitinBot
59Disallow: /
60
61Sitemap: https://www.newsweek.com/sitemap.xml
62Sitemap: https://www.newsweek.com/googlenews.xml
63Sitemap: https://www.newsweek.com/sitemap-2026.xml
64Sitemap: https://www.newsweek.com/sitemap-2025.xml
65Sitemap: https://www.newsweek.com/sitemap-2024.xml
66Sitemap: https://www.newsweek.com/sitemap-2023.xml
67Sitemap: https://www.newsweek.com/sitemap-2022.xml
68Sitemap: https://www.newsweek.com/sitemap-2021.xml
69Sitemap: https://www.newsweek.com/sitemap-2020.xml
70Sitemap: https://www.newsweek.com/sitemap-2019.xml
71Sitemap: https://www.newsweek.com/sitemap-2018.xml
72Sitemap: https://www.newsweek.com/sitemap-2017.xml
73Sitemap: https://www.newsweek.com/sitemap-2016.xml
74Sitemap: https://www.newsweek.com/sitemap-2015.xml
75Sitemap: https://www.newsweek.com/sitemap-2014.xml
76Sitemap: https://www.newsweek.com/sitemap-2013.xml
77Sitemap: https://www.newsweek.com/sitemap-2012.xml
78Sitemap: https://www.newsweek.com/sitemap-2011.xml
79Sitemap: https://www.newsweek.com/sitemap-2010.xml
80Sitemap: https://www.newsweek.com/sitemap-2009.xml
81Sitemap: https://www.newsweek.com/sitemap-2008.xml
82Sitemap: https://www.newsweek.com/sitemap-2007.xml
83Sitemap: https://www.newsweek.com/sitemap-2006.xml
84Sitemap: https://www.newsweek.com/sitemap-2005.xml
85Sitemap: https://www.newsweek.com/sitemap-2004.xml
86Sitemap: https://www.newsweek.com/sitemap-2003.xml
87Sitemap: https://www.newsweek.com/sitemap-2002.xml
88Sitemap: https://www.newsweek.com/sitemap-2001.xml
89Sitemap: https://www.newsweek.com/sitemap-2000.xml
90Sitemap: https://www.newsweek.com/sitemap-1999.xml
91Sitemap: https://www.newsweek.com/sitemap-1998.xml
92Sitemap: https://www.newsweek.com/sitemap-1997.xml
93Sitemap: https://www.newsweek.com/sitemap-1996.xml
94Sitemap: https://www.newsweek.com/sitemap-1995.xml
95Sitemap: https://www.newsweek.com/sitemap-1994.xml
96Sitemap: https://www.newsweek.com/sitemap-1993.xml
97Sitemap: https://www.newsweek.com/sitemap-1992.xml
98Sitemap: https://www.newsweek.com/sitemap-1991.xml
99Sitemap: https://www.newsweek.com/sitemap-1990.xml
100Sitemap: https://www.newsweek.com/sitemap-1988.xml
101Sitemap: https://www.newsweek.com/sitemap-1987.xml
102Sitemap: https://www.newsweek.com/sitemap-1986.xml
103Sitemap: https://www.newsweek.com/sitemap-1985.xml
104Sitemap: https://www.newsweek.com/sitemap-1984.xml
105Sitemap: https://www.newsweek.com/sitemap-1983.xml
106Sitemap: https://www.newsweek.com/sitemap-1982.xml
107Sitemap: https://www.newsweek.com/sitemap-1981.xml
108Sitemap: https://www.newsweek.com/sitemap-1980.xml
109Sitemap: https://www.newsweek.com/sitemap-1979.xml
110Sitemap: https://www.newsweek.com/sitemap-1975.xml
111Sitemap: https://www.newsweek.com/sitemap-1970.xml
112Sitemap: https://www.newsweek.com/sitemap-1969.xml
113Sitemap: https://www.newsweek.com/sitemap-1968.xml
114Sitemap: https://www.newsweek.com/sitemap-1964.xml
115Sitemap: https://www.newsweek.com/sitemap-1961.xml