NeuralCrawl

Toll Brothers / robots.txt snapshot

← back to tollbrothers.com · fetched 2026-06-20T01:10:30Z (18h ago) · HTTP 200 · 2314 bytes · sha256 2c018fcc093f6c86 · raw

final URL: https://www.tollbrothers.com/robots.txt

1Sitemap: https://www.tollbrothers.com/sitemap.xml
2
3User-agent: Aboundexbot
4Disallow: /
5
6User-agent: AhrefsBot
7Disallow: /
8
9User-agent: aiHitBot
10Disallow: /
11
12User-agent: BLEXBot
13Disallow: /
14
15User-agent: EasouSpider
16Disallow: /
17
18User-agent: EC2LinkFinder
19Disallow: /
20
21User-agent: Exabot
22Disallow: /
23
24User-agent: Ezooms
25Disallow: /
26
27User-agent: InfoPath
28Disallow: /
29
30User-agent: InfoPath.2
31Disallow: /
32
33User-agent: ip-web-crawler.com
34Disallow: /
35
36User-agent: LinkpadBot
37Disallow: /
38
39User-agent: Mail.Ru
40Disallow: /
41
42User-agent: meanpathbot
43Disallow: /
44
45User-agent: MJ12bot
46Disallow: /
47
48User-agent: netEstate NE Crawler
49Disallow: /
50
51User-agent: panscient.com
52Disallow: /
53
54User-agent: psbot
55Disallow: /
56
57User-agent: SearchmetericsBot
58Disallow: /
59
60User-agent: SeznamBot
61Disallow: /
62
63User-agent: sistrix
64Disallow: /
65
66User-agent: sitebot
67Disallow: /
68
69User-agent: Sosospider
70Disallow: /
71
72User-agent: spbot
73Disallow: /
74
75User-agent: swebot
76Disallow: /
77
78User-agent: taptubot
79Disallow: /
80
81User-agent: TurnitinBot
82Disallow: /
83
84User-agent: TwengaBot
85Disallow: /
86
87User-agent: twiceler
88Disallow: /
89
90User-agent: vscooter
91Disallow: /
92
93User-agent: WBSearchBot
94Disallow: /
95
96User-agent: wotbox
97Disallow: /
98
99User-agent: Yandex
100Disallow: /
101
102
103User-agent: *
104Disallow: /about/your-story
105Disallow: /about/your-story?*
106Disallow: /agent/rest/add/user/connect
107Disallow: /agent/rest/add/user/new
108Disallow: /agent/rest/login
109Disallow: /best-offer/thank-you
110Disallow: /best-offer/thank-you.php
111Disallow: /blog/wp-admin/
112Disallow: /cgi-bin/
113Disallow: /compare
114Disallow: /compare?*
115Disallow: /contact/mortgageSubmit
116Disallow: /external/
117Disallow: /external/*
118Disallow: /highrise
119Disallow: /highrise/
120Disallow: /highrise/*
121Disallow: /lead/process
122Disallow: /luxury-homes/Edgewood-TX
123Disallow: /luxury-homes/Edgewood-TX?*
124Disallow: /luxury-homes-for-sale/*/siteplans/*
125Disallow: /my-favorites/offpagelogin
126Disallow: /my_favorites/offpagelogin
127Disallow: /my-favorites/processUnsubscribe
128Disallow: /my_favorites/processUnsubscribe
129Disallow: /new-home-journey
130Disallow: /new-home-journey?*
131Disallow: /new-homes
132Disallow: /new-homes?*
133Disallow: /nse
134Disallow: /nse?*
135Disallow: /print
136Disallow: /print?*
137Disallow: /service*
138Disallow: /sitesearch
139Disallow: /sitesearch?*
140Disallow: /survey/
141Disallow: /tb/about/index.html
142Allow: /blog/wp-admin/admin-ajax.php