NeuralCrawl

GitHub / robots.txt

← back to github.com · change detected 7d ago (2026-09-12T01:21:55Z) · modified +140 −9

What changed

Diff old (2026-06-20T01:10:31Z) → new (2026-09-12T01:21:55Z)

@@ -1,5 +1,144 @@
11  # If you would like to crawl GitHub contact us via https://support.github.com?tags=dotcom-robots
22  # We also provide an extensive API: https://docs.github.com
3+ User-agent: GPTBot
4+ User-agent: OAI-SearchBot
5+ User-agent: ClaudeBot
6+ User-agent: anthropic-ai
7+ User-agent: PerplexityBot
8+ Crawl-delay: 1
9+ Allow: /$
10+ Allow: /about
11+ Allow: /customer-stories
12+ Allow: /enterprise
13+ Allow: /features
14+ Allow: /marketplace
15+ Allow: /mcp
16+ Allow: /newsroom
17+ Allow: /pricing
18+ Allow: /resources
19+ Allow: /security
20+ Allow: /solutions
21+ Disallow: /*/*/pulse
22+ Disallow: /*/*/projects
23+ Disallow: /*/*/forks
24+ Disallow: /*/*/issues/new
25+ Disallow: /*/*/milestones/new
26+ Disallow: /*/*/issues/search
27+ Disallow: /*/*/commits/
28+ Disallow: /*/*/branches
29+ Disallow: /*/*/contributors
30+ Disallow: /*/*/tags
31+ Disallow: /*/*/stargazers
32+ Disallow: /*/*/watchers
33+ Disallow: /*/*/network
34+ Disallow: /*/*/graphs
35+ Disallow: /*/*/compare
36+ Disallow: /*/tree/
37+ Disallow: /gist/
38+ Disallow: /*/download
39+ Disallow: /*/revisions
40+ Disallow: /*/commits/*?author
41+ Disallow: /*/commits/*?path
42+ Disallow: /*/comments
43+ Disallow: /*/archive/
44+ Disallow: /*/blame/
45+ Disallow: /*/raw/
46+ Disallow: /*/cache/
47+ Disallow: /.git/
48+ Disallow: */.git/
49+ Disallow: /*.git$
50+ Disallow: /search/advanced
51+ Disallow: /search$
52+ Disallow: /*q=
53+ Disallow: /*.atom$
54+ Disallow: /ekansa/Open-Context-Data
55+ Disallow: /ekansa/opencontext-*
56+ Disallow: */tarball/
57+ Disallow: */zipball/
58+ Disallow: /*source=*
59+ Disallow: /*ref_cta=*
60+ Disallow: /*plan=*
61+ Disallow: /*return_to=*
62+ Disallow: /*ref_loc=*
63+ Disallow: /*setup_organization=*
64+ Disallow: /*source_repo=*
65+ Disallow: /*ref_page=*
66+ Disallow: /*referrer=*
67+ Disallow: /*report=*
68+ Disallow: /*author=*
69+ Disallow: /*since=*
70+ Disallow: /*until=*
71+ Disallow: /*commits?author=*
72+ Disallow: /*report-abuse?report=*
73+ Disallow: /*tab=*
74+ Disallow: /account-login
75+ Disallow: /Explodingstuff/
76+ Disallow: /copilot/
77+ Disallow: /copilot/c/
78+
79+ User-agent: Bytespider
80+ Disallow: /
81+
82+ User-agent: AdsBot-Google
83+ User-agent: AdsBot-Google-Mobile
84+ Disallow: /*/*/pulse
85+ Disallow: /*/*/projects
86+ Disallow: /*/*/forks
87+ Disallow: /*/*/issues/new
88+ Disallow: /*/*/milestones/new
89+ Disallow: /*/*/issues/search
90+ Disallow: /*/*/commits/
91+ Disallow: /*/*/branches
92+ Disallow: /*/*/contributors
93+ Disallow: /*/*/tags
94+ Disallow: /*/*/stargazers
95+ Disallow: /*/*/watchers
96+ Disallow: /*/*/network
97+ Disallow: /*/*/graphs
98+ Disallow: /*/*/compare
99+ Disallow: /*/tree/
100+ Disallow: /gist/
101+ Disallow: /*/download
102+ Disallow: /*/revisions
103+ Disallow: /*/commits/*?author
104+ Disallow: /*/commits/*?path
105+ Disallow: /*/comments
106+ Disallow: /*/archive/
107+ Disallow: /*/blame/
108+ Disallow: /*/raw/
109+ Disallow: /*/cache/
110+ Disallow: /.git/
111+ Disallow: */.git/
112+ Disallow: /*.git$
113+ Disallow: /search/advanced
114+ Disallow: /search$
115+ Disallow: /*q=
116+ Disallow: /*.atom$
117+ Disallow: /ekansa/Open-Context-Data
118+ Disallow: /ekansa/opencontext-*
119+ Disallow: */tarball/
120+ Disallow: */zipball/
121+ Disallow: /*source=*
122+ Disallow: /*ref_cta=*
123+ Disallow: /*plan=*
124+ Disallow: /*return_to=*
125+ Disallow: /*ref_loc=*
126+ Disallow: /*setup_organization=*
127+ Disallow: /*source_repo=*
128+ Disallow: /*ref_page=*
129+ Disallow: /*referrer=*
130+ Disallow: /*report=*
131+ Disallow: /*author=*
132+ Disallow: /*since=*
133+ Disallow: /*until=*
134+ Disallow: /*commits?author=*
135+ Disallow: /*report-abuse?report=*
136+ Disallow: /*tab=*
137+ Disallow: /account-login
138+ Disallow: /Explodingstuff/
139+ Disallow: /copilot/
140+ Disallow: /copilot/c/
141+
3142  User-agent: bingbot
4143  Disallow: /ekansa/Open-Context-Data
5144  Disallow: /ekansa/opencontext-*
@@ -35,7 +174,7 @@
35174 
36175 
37176  User-agent: *
38-
177+ Allow: /*?tab=achievements&achievement=*
39178  Disallow: /*/*/pulse
40179  Disallow: /*/*/projects
41180  Disallow: /*/*/forks
@@ -51,7 +190,6 @@
51190  Disallow: /*/*/network
52191  Disallow: /*/*/graphs
53192  Disallow: /*/*/compare
54-
55193  Disallow: /*/tree/
56194  Disallow: /gist/
57195  Disallow: /*/download
@@ -70,12 +208,10 @@
70208  Disallow: /search$
71209  Disallow: /*q=
72210  Disallow: /*.atom$
73-
74211  Disallow: /ekansa/Open-Context-Data
75212  Disallow: /ekansa/opencontext-*
76213  Disallow: */tarball/
77214  Disallow: */zipball/
78-
79215  Disallow: /*source=*
80216  Disallow: /*ref_cta=*
81217  Disallow: /*plan=*
@@ -84,7 +220,6 @@
84220  Disallow: /*setup_organization=*
85221  Disallow: /*source_repo=*
86222  Disallow: /*ref_page=*
87- Disallow: /*source=*
88223  Disallow: /*referrer=*
89224  Disallow: /*report=*
90225  Disallow: /*author=*
@@ -93,11 +228,7 @@
93228  Disallow: /*commits?author=*
94229  Disallow: /*report-abuse?report=*
95230  Disallow: /*tab=*
96- Allow: /*?tab=achievements&achievement=*
97-
98231  Disallow: /account-login
99232  Disallow: /Explodingstuff/
100-
101233  Disallow: /copilot/
102234  Disallow: /copilot/c/
103-