NeuralCrawl

Indeed / robots.txt

← back to indeed.com · change detected 20h ago (2026-09-18T15:27:37Z) · modified +33 −17

What changed

Diff old (2026-08-18T05:21:46Z) → new (2026-09-18T15:27:37Z)

@@ -1,3 +1,4 @@
1+ # Default rules for all unspecified bots. Note that these apply to Bingbot too.
12  User-agent: *
23  Allow: /
34  Allow: /*&start=0&
@@ -57,6 +58,7 @@
5758  Disallow: /empregos/title
5859  Disallow: /forum/profile
5960  Disallow: /g/
61+ Disallow: /generate-question
6062  Disallow: /graphql
6163  Disallow: /imgping
6264  Disallow: /ita?
@@ -118,7 +120,7 @@
118120  Disallow: /m/newjobs
119121  Disallow: /m/recommended
120122  Disallow: /m/rpc/
121- Disallow: /m/viewjob?
123+ Disallow: /m/viewjob
122124  Disallow: /my/
123125  Disallow: /*oc=1
124126  Disallow: /ofertas/ES/
@@ -160,24 +162,29 @@
160162  Disallow: /trabajo/
161163  Disallow: /uie/clk
162164  Disallow: /url
165+ Disallow: /v1/alerts
166+ Disallow: /v1/impressions
163167  Disallow: /vacature/
164- Disallow: /vacature-bekijken?
168+ Disallow: /vacature-bekijken
165169  Disallow: /vacatures/NL/
166170  Disallow: /vacatures/title/
167- Disallow: /ver-empleo?
168- Disallow: /ver-emprego?
169- Disallow: /ver-oferta?
170- Disallow: /viewjob?
171- Disallow: /visajobb?
172- Disallow: /voir-emploi?
173- Disallow: /Zeige-Job?
171+ Disallow: /ver-empleo
172+ Disallow: /ver-emprego
173+ Disallow: /ver-oferta
174+ Disallow: /viewjob
175+ Disallow: /visajobb
176+ Disallow: /voir-emploi
177+ Disallow: /Zeige-Job
174178  Disallow: //applystart
179+ Disallow: /hire/o/ats-partners
180+ Disallow: /hire/ats-partners
181+ Disallow: /offers
175182 
176183 
184+ # Rules for traditional search engines and for retrieval/grounding bots.
177185  User-agent: Googlebot
178186  User-agent: Google-Extended
179187  User-agent: ChatGPT-User
180- User-agent: Bingbot
181188  User-agent: Slurp
182189  User-agent: DuckDuckBot
183190  User-agent: YahooJP
@@ -387,6 +394,7 @@
387394  Disallow: /trendgraph/*
388395 
389396 
397+ # Full disallow for aggressive scrapers, legacy crawlers, and unwanted utility bots.
390398  User-Agent: OmniExplorer_Bot
391399  User-agent: ia_archiver
392400  User-agent: 008
@@ -402,6 +410,7 @@
402410  Disallow: /
403411 
404412 
413+ # Rules for Social Media / Link Preview bots.
405414  User-agent: facebookexternalhit
406415  User-agent: WhatsApp
407416  User-agent: snapchat
@@ -411,6 +420,7 @@
411420  Allow: /
412421 
413422 
423+ # Rules for Foundational (Training) bots.
414424  User-agent: GPTBot
415425  User-agent: CCBot
416426  User-agent: anthropic-ai
@@ -474,6 +484,7 @@
474484  Disallow: /emprego/
475485  Disallow: /forum/profile
476486  Disallow: /g/
487+ Disallow: /generate-question
477488  Disallow: /graphql
478489  Disallow: /imgping
479490  Disallow: /ita?
@@ -526,14 +537,19 @@
526537  Disallow: /tos/banner.js
527538  Disallow: /uie/clk
528539  Disallow: /url
540+ Disallow: /v1/alerts
541+ Disallow: /v1/impressions
529542  Disallow: /vacature/
530- Disallow: /vacature-bekijken?
531- Disallow: /ver-empleo?
532- Disallow: /ver-emprego?
533- Disallow: /ver-oferta?
534- Disallow: /visajobb?
535- Disallow: /voir-emploi?
536- Disallow: /Zeige-Job?
543+ Disallow: /vacature-bekijken
544+ Disallow: /ver-empleo
545+ Disallow: /ver-emprego
546+ Disallow: /ver-oferta
547+ Disallow: /visajobb
548+ Disallow: /voir-emploi
549+ Disallow: /Zeige-Job
550+ Disallow: /hire/o/ats-partners
551+ Disallow: /hire/ats-partners
552+ Disallow: /offers
537553  Disallow: /c/info/
538554  Disallow: /hire/how-to-hire/
539555  Disallow: /hire/interview-questions/