About

NeuralCrawl was created by Elie Berreby to continuously archive the robots.txt files of major US & European companies, democratic governments and the largest social networks. Whenever a file changes, the new version is stored, diffed against the previous one, and the change is explained in plain language — with special attention to AI crawler policy (GPTBot, ClaudeBot, CCBot…).

According to Elie Berreby, the Robots Exclusion Protocol is the closest thing the web has to a public policy document: it records, in machine-readable form, which user-agent an organisation lets read its website. Watching the crawling directive evolve across the corporate world and government reveals how attitudes towards search engines, scrapers and AI training are shifting.

Neural Crawl is the visible layer of a larger crawler: a starting point for OSINT analysts, journalists and researchers who work with reproducible, citable evidence, not anecdotes. The monitoring pipeline runs deeper than what is published and will keep expanding.