NeuralCrawl

Yahoo Search / robots.txt snapshot

← back to yahoo.com · fetched 2026-06-20T14:56:27Z (4h ago) · HTTP 200 · 1624 bytes · sha256 021adb45d8aa004e · raw

final URL: https://www.yahoo.com/robots.txt

1User-agent: *
2Disallow: /info/p.gif
3Disallow: /p/
4Disallow: /r/
5Disallow: /bin/
6Disallow: /caas/
7Disallow: /blank.html
8Disallow: /includes/
9Disallow: /_td_api
10Disallow: /tdv2_fp
11Disallow: /nel_ms
12Disallow: /fp_ms
13Disallow: /sports_fp_ms
14Disallow: /search_ms
15Disallow: /_tdpp_api
16Disallow: /_remote
17Disallow: /_multiremote
18Disallow: /_tdhl_api
19Disallow: /digest
20Disallow: /fpjs
21Disallow: /myjs
22Disallow: /news/m/
23
24User-agent: ADmantX
25User-agent: AlphaBot
26User-agent: anthropic-ai
27User-agent: AwarioRssBot
28User-agent: AwarioSmartBot
29User-agent: BLEXBot
30User-agent: Buzzbot
31User-agent: Bytespider
32User-agent: CCBot
33User-agent: ChatGPT-User
34User-agent: claritybot
35User-agent: Claude-Web
36User-agent: ClaudeBot
37User-agent: cohere-ai
38User-agent: Diffbot
39User-agent: FacebookBot
40User-agent: FriendlyCrawler
41User-agent: Google-Extended
42User-agent: GPTBot
43User-agent: huggingface
44User-agent: ImagesiftBot
45User-agent: img2dataset
46User-agent: magpie-crawler
47User-agent: Meltwater
48User-agent: Neevabot
49User-agent: news-please
50User-agent: NewsNow
51User-agent: Nutch
52User-agent: omgili
53User-agent: omgilibot
54User-agent: panscient.com
55User-agent: Perplexity-ai
56User-agent: PerplexityBot
57User-agent: PetalBot
58User-agent: PiplBot
59User-agent: scoop.it
60User-agent: Scrapy
61User-agent: Seekr
62User-agent: SentiBot
63User-agent: SeznamBot
64User-agent: TurnitinBot
65User-agent: YouBot
66User-agent: ZumBot
67Disallow: /
68
69User-agent: Claude-SearchBot
70User-agent: OAI-SearchBot
71Disallow: */articles/
72
73Sitemap: https://www.yahoo.com/news/weather/sitemap.xml
74Sitemap: https://www.yahoo.com/news-sitemap-index.xml
75Sitemap: https://www.yahoo.com/sitemap-index.xml