NeuralCrawl

GitHub / robots.txt

← back to github.com · change detected 1mo ago (2026-10-02T01:50:39Z) · modified +56 −0

What changed

Diff old (2026-09-30T19:18:16Z) → new (2026-10-02T01:50:39Z)

@@ -63,6 +63,24 @@
6363  Disallow: /Explodingstuff/
6464  Disallow: /copilot/
6565  Disallow: /copilot/c/
66+ Disallow: /marketplace/*/*?tab=readme$
67+ Disallow: /marketplace/*?*?tab=readme$
68+ Disallow: /marketplace/actions?tab=readme$
69+ Disallow: /marketplace/free-trials?tab=readme$
70+ Disallow: /marketplace/manage?tab=readme$
71+ Disallow: /marketplace/models?tab=readme$
72+ Disallow: /marketplace/new?tab=readme$
73+ Disallow: /marketplace/publishers?tab=readme$
74+ Disallow: /marketplace/recently-added?tab=readme$
75+ Disallow: /marketplace/*/*?tab=transparency$
76+ Disallow: /marketplace/*?*?tab=transparency$
77+ Disallow: /marketplace/actions?tab=transparency$
78+ Disallow: /marketplace/free-trials?tab=transparency$
79+ Disallow: /marketplace/manage?tab=transparency$
80+ Disallow: /marketplace/models?tab=transparency$
81+ Disallow: /marketplace/new?tab=transparency$
82+ Disallow: /marketplace/publishers?tab=transparency$
83+ Disallow: /marketplace/recently-added?tab=transparency$
6684  Disallow: /*?source=*
6785  Disallow: /*&source=*
6886  Disallow: /*?ref_cta=*
@@ -231,6 +249,24 @@
231249  Disallow: /Explodingstuff/
232250  Disallow: /copilot/
233251  Disallow: /copilot/c/
252+ Disallow: /marketplace/*/*?tab=readme$
253+ Disallow: /marketplace/*?*?tab=readme$
254+ Disallow: /marketplace/actions?tab=readme$
255+ Disallow: /marketplace/free-trials?tab=readme$
256+ Disallow: /marketplace/manage?tab=readme$
257+ Disallow: /marketplace/models?tab=readme$
258+ Disallow: /marketplace/new?tab=readme$
259+ Disallow: /marketplace/publishers?tab=readme$
260+ Disallow: /marketplace/recently-added?tab=readme$
261+ Disallow: /marketplace/*/*?tab=transparency$
262+ Disallow: /marketplace/*?*?tab=transparency$
263+ Disallow: /marketplace/actions?tab=transparency$
264+ Disallow: /marketplace/free-trials?tab=transparency$
265+ Disallow: /marketplace/manage?tab=transparency$
266+ Disallow: /marketplace/models?tab=transparency$
267+ Disallow: /marketplace/new?tab=transparency$
268+ Disallow: /marketplace/publishers?tab=transparency$
269+ Disallow: /marketplace/recently-added?tab=transparency$
234270  Disallow: /*?source=*
235271  Disallow: /*&source=*
236272  Disallow: /*?ref_cta=*
@@ -298,6 +334,8 @@
298334 
299335  User-agent: *
300336  Allow: /*?tab=achievements&achievement=*
337+ Allow: /marketplace/*?tab=readme$
338+ Allow: /marketplace/*?tab=transparency$
301339  Allow: /topics/download*
302340  Allow: /topics/comments*
303341  Allow: /marketplace/actions/download*
@@ -343,6 +381,24 @@
343381  Disallow: /Explodingstuff/
344382  Disallow: /copilot/
345383  Disallow: /copilot/c/
384+ Disallow: /marketplace/*/*?tab=readme$
385+ Disallow: /marketplace/*?*?tab=readme$
386+ Disallow: /marketplace/actions?tab=readme$
387+ Disallow: /marketplace/free-trials?tab=readme$
388+ Disallow: /marketplace/manage?tab=readme$
389+ Disallow: /marketplace/models?tab=readme$
390+ Disallow: /marketplace/new?tab=readme$
391+ Disallow: /marketplace/publishers?tab=readme$
392+ Disallow: /marketplace/recently-added?tab=readme$
393+ Disallow: /marketplace/*/*?tab=transparency$
394+ Disallow: /marketplace/*?*?tab=transparency$
395+ Disallow: /marketplace/actions?tab=transparency$
396+ Disallow: /marketplace/free-trials?tab=transparency$
397+ Disallow: /marketplace/manage?tab=transparency$
398+ Disallow: /marketplace/models?tab=transparency$
399+ Disallow: /marketplace/new?tab=transparency$
400+ Disallow: /marketplace/publishers?tab=transparency$
401+ Disallow: /marketplace/recently-added?tab=transparency$
346402  Disallow: /*?source=*
347403  Disallow: /*&source=*
348404  Disallow: /*?ref_cta=*