NeuralCrawl

Indeed / robots.txt

← back to indeed.com · change detected 17d ago (2026-06-30T18:10:20Z) · modified +30 −177

What changed

Diff old (2026-06-26T23:05:56Z) → new (2026-06-30T18:10:20Z)

@@ -179,6 +179,13 @@
179179  User-agent: Claude-User
180180  User-agent: Gemini-Deep-Research
181181  User-agent: Claude-SearchBot
182+ User-agent: OAI-SearchBot
183+ User-agent: Google-Agent
184+ User-agent: Google-CloudVertexBot
185+ User-agent: Meta-WebIndexer
186+ User-agent: Meta-ExternalFetcher
187+ User-agent: xAI-Grok
188+ User-agent: Grok-DeepSearch
182189  Allow: /
183190  Disallow: /*rt=nc
184191  Disallow: /*&alid=
@@ -340,174 +347,9 @@
340347  Disallow: /offers
341348 
342349 
343-
344- User-agent: OAI-SearchBot
345- Allow: /
346- Disallow: /*rt=nc
347- Disallow: /*&alid=
348- Disallow: /*&calert=
349- Disallow: /*&iafilter=
350- Disallow: /*&mna=
351- Disallow: /*?rss
352- Disallow: /*&serpstart=
353- Disallow: /*/vaclk
354- Disallow: /addlLoc/
355- Disallow: /ads/
356- Disallow: /advanced_search
357- Disallow: /alert
358- Disallow: /api/fetch/mc-anon
359- Disallow: /api/getrecjobs
360- Disallow: /applystart
361- Disallow: /cmp/_/
362- Disallow: /cmp/_c/claim/*
363- Disallow: /cmp/_rpc/
364- Disallow: /cmp/addlink
365- Disallow: /cmp/addvideo
366- Disallow: /cmp/Login*
367- Disallow: /cmp/*/analytics
368- Disallow: /cmp/*/company-questions
369- Disallow: /cmp/*/people
370- Disallow: /cmp/*/write-review
371- Disallow: /community/
372- Disallow: /company/*
373- Disallow: /conversion/
374- Disallow: /cookiemigrator/
375- Disallow: /create-resume/lp/
376- Disallow: /cdn-cgi/
377- Disallow: /%E4%BB%95%E4%BA%8B?
378- Disallow: /%E5%B7%A5%E4%BD%9C/
379- Disallow: /%E6%B1%82%E4%BA%BA/JP/
380- Disallow: /%E6%B1%82%E4%BA%BA/title/
381- Disallow: /%E8%81%8C%E4%BD%8D%E6%98%BE%E7%A4%BA?
382- Disallow: /%EC%B7%A8%EC%97%85/KR/
383- Disallow: /%EC%B7%A8%EC%97%85/title/
384- Disallow: /empleo/
385- Disallow: /emploi/
386- Disallow: /emplois/FR/
387- Disallow: /emplois/title/
388- Disallow: /emprego/
389- Disallow: /empregos/BR/
390- Disallow: /empregos/title
391- Disallow: /forum/profile
392- Disallow: /g/
393- Disallow: /generate-question
394- Disallow: /graphql
395- Disallow: /imgping
396- Disallow: /ita?
397- Disallow: /ita/v1/publisher
398- Disallow: /ja/clk?
399- Disallow: /ja/imp.gif
400- Disallow: /job/
401- Disallow: /Job/
402- Disallow: /jobb/
403- Disallow: /jobroll
404- Disallow: /jobs/AE/
405- Disallow: /jobs/AQ/
406- Disallow: /Jobs/AT/
407- Disallow: /jobs/AU/
408- Disallow: /jobs/BE/
409- Disallow: /jobs/BH/
410- Disallow: /jobs/CA/
411- Disallow: /jobs/CZ/
412- Disallow: /jobs/DE/
413- Disallow: /Jobs/DE/
414- Disallow: /jobs/DK/
415- Disallow: /jobs/FI/
416- Disallow: /jobs/GB/
417- Disallow: /jobs/GR/
418- Disallow: /jobs/HK/
419- Disallow: /jobs/HU/
420- Disallow: /jobs/ID/
421- Disallow: /jobs/IE/
422- Disallow: /jobs/IL/
423- Disallow: /jobs/IN/
424- Disallow: /jobs/KW/
425- Disallow: /jobs/LU/
426- Disallow: /jobs/MY/
427- Disallow: /jobs/NO/
428- Disallow: /jobs/NZ/
429- Disallow: /jobs/OM/
430- Disallow: /jobs/PE/
431- Disallow: /jobs/PH/
432- Disallow: /jobs/PK/
433- Disallow: /jobs/PT/
434- Disallow: /jobs/QA/
435- Disallow: /jobs/RO/
436- Disallow: /jobs/RU/
437- Disallow: /jobs/SA/
438- Disallow: /jobs/SG/
439- Disallow: /jobs/title
440- Disallow: /Jobs/title
441- Disallow: /jobs/TR/
442- Disallow: /jobs/TW/
443- Disallow: /jobs/US/
444- Disallow: /jobs/VE/
445- Disallow: /jobs/ZA/
446- Disallow: /jobtrends/trends/log/*
447- Disallow: /jwidget.js*
448- Disallow: /me/*/pdf
449- Disallow: /m/basecamp/analytics.js
450- Disallow: /m/jobalerts?
451- Disallow: /m/moreLoc
452- Disallow: /m/newjobs
453- Disallow: /m/recommended
454- Disallow: /m/rpc/
455- Disallow: /my/
456- Disallow: /*oc=1
457- Disallow: /ofertas/ES/
458- Disallow: /ofertas/title/
459- Disallow: /offerte-lavoro/IT/
460- Disallow: /offerte-lavoro/title
461- Disallow: /pagead/
462- Disallow: /poka%C5%BCprac%C4%99?
463- Disallow: /praca/
464- Disallow: /preferences
465- Disallow: /*radius=
466- Disallow: /rc/
467- Disallow: /rdr/
468- Disallow: /recommendedjobs
469- Disallow: /record-page-tags
470- Disallow: /resumes/account
471- Disallow: /resumes/advanced?
472- Disallow: /resumes/alert
473- Disallow: /resumes*radius=
474- Disallow: /resumes*rb=
475- Disallow: /resumes*res=
476- Disallow: /resumes/rpc/
477- Disallow: /resumes*sort=
478- Disallow: /resumes*start=
479- Disallow: /rpc/
480- Disallow: /r/*/pdf
481- Disallow: /rss
482- Disallow: /setprefs
483- Disallow: /*sp=0
484- Disallow: /*&start=
485- Disallow: /stc
486- Disallow: /Stellen/CH/
487- Disallow: /Stellen/title
488- Disallow: /tellafriend
489- Disallow: /tmn/ccs/
490- Disallow: /tos/banner.js
491- Disallow: /trabajo/
492- Disallow: /trabajo/title/
493- Disallow: /uie/clk
494- Disallow: /url
495- Disallow: /v1/alerts
496- Disallow: /v1/impressions
497- Disallow: /vacature/
498- Disallow: /vacatures/NL/
499- Disallow: /vacatures/title/
500- Disallow: //applystart
501-
502-
503- User-agent: ia_archiver
504- Disallow: /
505-
506- User-Agent: OmniExplorer_Bot
507- Disallow: /
508-
509350  User-agent: Mediapartners-Google
510351  Allow: /
352+
511353 
512354  User-agent: AdsBot-Google
513355  User-agent: AdsBot-Google-Mobile
@@ -515,33 +357,39 @@
515357  Allow: /*sid=
516358  Disallow: /rpc/
517359 
360+
518361  User-agent: adidxbot
519362  Allow: /*sid=
520363 
364+
521365  User-agent: Googlebot-Image
522- Disallow: /trendgraph/*
523-
524366  User-agent: msnbot-media
525367  Disallow: /trendgraph/*
526368 
369+
370+ User-Agent: OmniExplorer_Bot
371+ User-agent: ia_archiver
527372  User-agent: 008
528- Disallow: /
373+ User-agent: JobdiggerSpider
374+ User-agent: Cliqzbot
375+ User-agent: GoogleOther
376+ User-agent: omgili
377+ User-agent: omgilibot
378+ User-agent: Webzio
379+ User-agent: Webzio-Extended
380+ User-agent: Scrapy
381+ User-agent: MyCentralAIScraperBot
382+ Disallow: /
529383 
530- User-agent: JobdiggerSpider
531- Disallow: /
532-
533- User-agent: Cliqzbot
534- Disallow: /
535-
536- User-agent: GoogleOther
537- Disallow: /
538384 
539385  User-agent: facebookexternalhit
540386  User-agent: WhatsApp
541387  User-agent: snapchat
542388  User-agent: TelegramBot
543389  User-agent: Twitterbot
390+ User-agent: Slackbot-LinkExpanding
544391  Allow: /
392+
545393 
546394  User-agent: GPTBot
547395  User-agent: CCBot
@@ -557,6 +405,11 @@
557405  User-agent: Meta-ExternalAgent
558406  User-agent: ClaudeBot
559407  User-agent: DeepSeekBot
408+ User-agent: GrokBot
409+ User-agent: Diffbot
410+ User-agent: AI2Bot
411+ User-agent: PanguBot
412+ User-agent: Timpibot
560413  Disallow: /*rt=nc
561414  Disallow: /*&alid=
562415  Disallow: /*&calert=