NeuralCrawl

Indeed / robots.txt

← back to indeed.com · change detected 1mo ago (2026-08-18T05:21:46Z) · modified +30 −0

What changed

Diff old (2026-06-30T18:10:20Z) → new (2026-08-18T05:21:46Z)

@@ -1,5 +1,15 @@
11  User-agent: *
22  Allow: /
3+ Allow: /*&start=0&
4+ Allow: /*&start=10&
5+ Allow: /*&start=20&
6+ Allow: /*&start=30&
7+ Allow: /*&start=40&
8+ Allow: /*&start=50&
9+ Allow: /*&start=60&
10+ Allow: /*&start=70&
11+ Allow: /*&start=80&
12+ Allow: /*&start=90&
313  Disallow: /*rt=nc
414  Disallow: /*&alid=
515  Disallow: /*&calert=
@@ -187,6 +197,16 @@
187197  User-agent: xAI-Grok
188198  User-agent: Grok-DeepSearch
189199  Allow: /
200+ Allow: /*&start=0&
201+ Allow: /*&start=10&
202+ Allow: /*&start=20&
203+ Allow: /*&start=30&
204+ Allow: /*&start=40&
205+ Allow: /*&start=50&
206+ Allow: /*&start=60&
207+ Allow: /*&start=70&
208+ Allow: /*&start=80&
209+ Allow: /*&start=90&
190210  Disallow: /*rt=nc
191211  Disallow: /*&alid=
192212  Disallow: /*&calert=
@@ -410,6 +430,16 @@
410430  User-agent: AI2Bot
411431  User-agent: PanguBot
412432  User-agent: Timpibot
433+ Allow: /*&start=0&
434+ Allow: /*&start=10&
435+ Allow: /*&start=20&
436+ Allow: /*&start=30&
437+ Allow: /*&start=40&
438+ Allow: /*&start=50&
439+ Allow: /*&start=60&
440+ Allow: /*&start=70&
441+ Allow: /*&start=80&
442+ Allow: /*&start=90&
413443  Disallow: /*rt=nc
414444  Disallow: /*&alid=
415445  Disallow: /*&calert=