NeuralCrawl

GitHub / robots.txt

← back to github.com · change detected 20h ago (2026-09-18T15:27:37Z) · modified +90 −45

What changed

Diff old (2026-09-12T01:21:55Z) → new (2026-09-18T15:27:37Z)

@@ -49,32 +49,47 @@
4949  Disallow: /*.git$
5050  Disallow: /search/advanced
5151  Disallow: /search$
52- Disallow: /*q=
5352  Disallow: /*.atom$
5453  Disallow: /ekansa/Open-Context-Data
5554  Disallow: /ekansa/opencontext-*
5655  Disallow: */tarball/
5756  Disallow: */zipball/
58- Disallow: /*source=*
59- Disallow: /*ref_cta=*
60- Disallow: /*plan=*
61- Disallow: /*return_to=*
62- Disallow: /*ref_loc=*
63- Disallow: /*setup_organization=*
64- Disallow: /*source_repo=*
65- Disallow: /*ref_page=*
66- Disallow: /*referrer=*
67- Disallow: /*report=*
68- Disallow: /*author=*
69- Disallow: /*since=*
70- Disallow: /*until=*
7157  Disallow: /*commits?author=*
7258  Disallow: /*report-abuse?report=*
73- Disallow: /*tab=*
7459  Disallow: /account-login
7560  Disallow: /Explodingstuff/
7661  Disallow: /copilot/
7762  Disallow: /copilot/c/
63+ Disallow: /*?source=*
64+ Disallow: /*&source=*
65+ Disallow: /*?ref_cta=*
66+ Disallow: /*&ref_cta=*
67+ Disallow: /*?plan=*
68+ Disallow: /*&plan=*
69+ Disallow: /*?return_to=*
70+ Disallow: /*&return_to=*
71+ Disallow: /*?ref_loc=*
72+ Disallow: /*&ref_loc=*
73+ Disallow: /*?setup_organization=*
74+ Disallow: /*&setup_organization=*
75+ Disallow: /*?source_repo=*
76+ Disallow: /*&source_repo=*
77+ Disallow: /*?ref_page=*
78+ Disallow: /*&ref_page=*
79+ Disallow: /*?referrer=*
80+ Disallow: /*&referrer=*
81+ Disallow: /*?report=*
82+ Disallow: /*&report=*
83+ Disallow: /*?author=*
84+ Disallow: /*&author=*
85+ Disallow: /*?since=*
86+ Disallow: /*&since=*
87+ Disallow: /*?until=*
88+ Disallow: /*&until=*
89+ Disallow: /*?tab=*
90+ Disallow: /*&tab=*
91+ Disallow: /*?q=*
92+ Disallow: /*&q=*
7893 
7994  User-agent: Bytespider
8095  Disallow: /
@@ -112,32 +127,47 @@
112127  Disallow: /*.git$
113128  Disallow: /search/advanced
114129  Disallow: /search$
115- Disallow: /*q=
116130  Disallow: /*.atom$
117131  Disallow: /ekansa/Open-Context-Data
118132  Disallow: /ekansa/opencontext-*
119133  Disallow: */tarball/
120134  Disallow: */zipball/
121- Disallow: /*source=*
122- Disallow: /*ref_cta=*
123- Disallow: /*plan=*
124- Disallow: /*return_to=*
125- Disallow: /*ref_loc=*
126- Disallow: /*setup_organization=*
127- Disallow: /*source_repo=*
128- Disallow: /*ref_page=*
129- Disallow: /*referrer=*
130- Disallow: /*report=*
131- Disallow: /*author=*
132- Disallow: /*since=*
133- Disallow: /*until=*
134135  Disallow: /*commits?author=*
135136  Disallow: /*report-abuse?report=*
136- Disallow: /*tab=*
137137  Disallow: /account-login
138138  Disallow: /Explodingstuff/
139139  Disallow: /copilot/
140140  Disallow: /copilot/c/
141+ Disallow: /*?source=*
142+ Disallow: /*&source=*
143+ Disallow: /*?ref_cta=*
144+ Disallow: /*&ref_cta=*
145+ Disallow: /*?plan=*
146+ Disallow: /*&plan=*
147+ Disallow: /*?return_to=*
148+ Disallow: /*&return_to=*
149+ Disallow: /*?ref_loc=*
150+ Disallow: /*&ref_loc=*
151+ Disallow: /*?setup_organization=*
152+ Disallow: /*&setup_organization=*
153+ Disallow: /*?source_repo=*
154+ Disallow: /*&source_repo=*
155+ Disallow: /*?ref_page=*
156+ Disallow: /*&ref_page=*
157+ Disallow: /*?referrer=*
158+ Disallow: /*&referrer=*
159+ Disallow: /*?report=*
160+ Disallow: /*&report=*
161+ Disallow: /*?author=*
162+ Disallow: /*&author=*
163+ Disallow: /*?since=*
164+ Disallow: /*&since=*
165+ Disallow: /*?until=*
166+ Disallow: /*&until=*
167+ Disallow: /*?tab=*
168+ Disallow: /*&tab=*
169+ Disallow: /*?q=*
170+ Disallow: /*&q=*
141171 
142172  User-agent: bingbot
143173  Disallow: /ekansa/Open-Context-Data
@@ -206,29 +236,44 @@
206236  Disallow: /*.git$
207237  Disallow: /search/advanced
208238  Disallow: /search$
209- Disallow: /*q=
210239  Disallow: /*.atom$
211240  Disallow: /ekansa/Open-Context-Data
212241  Disallow: /ekansa/opencontext-*
213242  Disallow: */tarball/
214243  Disallow: */zipball/
215- Disallow: /*source=*
216- Disallow: /*ref_cta=*
217- Disallow: /*plan=*
218- Disallow: /*return_to=*
219- Disallow: /*ref_loc=*
220- Disallow: /*setup_organization=*
221- Disallow: /*source_repo=*
222- Disallow: /*ref_page=*
223- Disallow: /*referrer=*
224- Disallow: /*report=*
225- Disallow: /*author=*
226- Disallow: /*since=*
227- Disallow: /*until=*
228244  Disallow: /*commits?author=*
229245  Disallow: /*report-abuse?report=*
230- Disallow: /*tab=*
231246  Disallow: /account-login
232247  Disallow: /Explodingstuff/
233248  Disallow: /copilot/
234249  Disallow: /copilot/c/
250+ Disallow: /*?source=*
251+ Disallow: /*&source=*
252+ Disallow: /*?ref_cta=*
253+ Disallow: /*&ref_cta=*
254+ Disallow: /*?plan=*
255+ Disallow: /*&plan=*
256+ Disallow: /*?return_to=*
257+ Disallow: /*&return_to=*
258+ Disallow: /*?ref_loc=*
259+ Disallow: /*&ref_loc=*
260+ Disallow: /*?setup_organization=*
261+ Disallow: /*&setup_organization=*
262+ Disallow: /*?source_repo=*
263+ Disallow: /*&source_repo=*
264+ Disallow: /*?ref_page=*
265+ Disallow: /*&ref_page=*
266+ Disallow: /*?referrer=*
267+ Disallow: /*&referrer=*
268+ Disallow: /*?report=*
269+ Disallow: /*&report=*
270+ Disallow: /*?author=*
271+ Disallow: /*&author=*
272+ Disallow: /*?since=*
273+ Disallow: /*&since=*
274+ Disallow: /*?until=*
275+ Disallow: /*&until=*
276+ Disallow: /*?tab=*
277+ Disallow: /*&tab=*
278+ Disallow: /*?q=*
279+ Disallow: /*&q=*