NeuralCrawl

Coursera / robots.txt

← back to coursera.org · change detected 1mo ago (2026-09-21T22:21:49Z) · modified +4 −11

What changed

Diff old (2026-09-11T00:58:05Z) → new (2026-09-21T22:21:49Z)

@@ -23,6 +23,10 @@
2323  Disallow: /business/teams/search
2424  Disallow: /organizations/
2525  Disallow: /programs/
26+ Disallow: /5JKLVK/
27+
28+ User-agent: AdsBot-Google
29+ Allow: /
2630 
2731  # Block all /courses pagination
2832  Disallow: /courses?page=
@@ -54,7 +58,6 @@
5458 
5559  Sitemap: https://www.coursera.org/sitemap.xml
5660 
57-
5861  User-agent: CCBot
5962  Disallow: /lecture/
6063  Disallow: /courses?page=
@@ -79,7 +82,6 @@
7982  Allow: /courses?*&page=4&
8083  Allow: /courses?*&page=5$
8184  Allow: /courses?*&page=5&
82-
8385 
8486  User-agent: GPTBot
8587  Disallow: /lecture/
@@ -106,7 +108,6 @@
106108  Allow: /courses?*&page=5$
107109  Allow: /courses?*&page=5&
108110 
109-
110111  User-agent: ClaudeBot
111112  Disallow: /lecture/
112113  Disallow: /courses?page=
@@ -131,7 +132,6 @@
131132  Allow: /courses?*&page=4&
132133  Allow: /courses?*&page=5$
133134  Allow: /courses?*&page=5&
134-
135135 
136136  User-agent: MistralAI-User
137137  Disallow: /lecture/
@@ -158,7 +158,6 @@
158158  Allow: /courses?*&page=5$
159159  Allow: /courses?*&page=5&
160160 
161-
162161  User-agent: Google-Extended
163162  Disallow: /lecture/
164163  Disallow: /courses?page=
@@ -183,7 +182,6 @@
183182  Allow: /courses?*&page=4&
184183  Allow: /courses?*&page=5$
185184  Allow: /courses?*&page=5&
186-
187185 
188186  User-agent: PerplexityBot
189187  Disallow: /lecture/
@@ -210,7 +208,6 @@
210208  Allow: /courses?*&page=5$
211209  Allow: /courses?*&page=5&
212210 
213-
214211  User-agent: LinkedInBot
215212  Allow: /account/accomplishments/
216213  Disallow: /courses?page=
@@ -235,7 +232,6 @@
235232  Allow: /courses?*&page=4&
236233  Allow: /courses?*&page=5$
237234  Allow: /courses?*&page=5&
238-
239235 
240236  User-agent: Twitterbot
241237  Allow: /account/accomplishments/
@@ -262,10 +258,8 @@
262258  Allow: /courses?*&page=5$
263259  Allow: /courses?*&page=5&
264260 
265-
266261  User-agent: meta-externalagent
267262  Disallow: /
268-
269263 
270264  User-agent: OAI-SearchBot
271265  Allow: /api/utilities/v1/imageproxy
@@ -293,7 +287,6 @@
293287  Allow: /courses?*&page=5$
294288  Allow: /courses?*&page=5&
295289 
296-
297290  User-agent: ChatGPT-User
298291  Allow: /api/utilities/v1/imageproxy
299292  Disallow: /api/