Screaming Frog / robots.txt snapshot
← back to screamingfrog.co.uk · fetched 2026-06-20T14:56:27Z (4h ago) · HTTP 200 · 1825 bytes · sha256 3f6d6469fae81c85 · raw
final URL: https://screamingfrog.co.uk/robots.txt
| 1 | User-agent: * |
| 2 | Disallow: /products/ |
| 3 | Disallow: /?s=* |
| 4 | Disallow: /*?post_type* |
| 5 | Disallow: /search/ |
| 6 | Disallow: /wp-content/themes/screamingfrog/inc/download-modal.php |
| 7 | Disallow: /wp-content/themes/screamingfrog/inc/download-modal-lfa.php |
| 8 | |
| 9 | # Protection of frog team |
| 10 | |
| 11 | User-agent: pesticides |
| 12 | Disallow: /frogs/ |
| 13 | |
| 14 | User-agent: toads |
| 15 | Disallow: /frogs/ |
| 16 | |
| 17 | User-agent: common salt |
| 18 | Disallow: /frogs/ |
| 19 | |
| 20 | User-agent: the french |
| 21 | Disallow: /frogs/legs/ |
| 22 | |
| 23 | # Protection of frog teams sanity |
| 24 | |
| 25 | User-agent: recruitment agencies |
| 26 | Disallow: /frogs/phones/ |
| 27 | |
| 28 | # Screaming Frog - Search Engine Marketing |
| 29 | |
| 30 | # If you're looking at our robots.txt then you might well be interested in our current SEO vacancies :-) |
| 31 | |
| 32 | # https://www.screamingfrog.co.uk/careers/ |
| 33 | |
| 34 | ........?$$$M$$$$$$$$$$$$$$$$$$......... |
| 35 | .. ..$$$MM$$M$M$$$$$$$$$N$$$$$$$. . .. |
| 36 | ....$$$$$$7MMM.M.......MMMMM$$$$$$$..... |
| 37 | ...$$$$$$. MMMM ...MMMMMMMMMM$$$$=.. |
| 38 | ..$$$$$.. MMM MMMMMMMMMMM.$$$$$I.. |
| 39 | .$$$$$... MMM$MMMMMMMMMMMMM. .$$$$$.. |
| 40 | $$$$$.... .$$MMMOMMMMMMMMMM .. ..$$$$$. |
| 41 | $$$$..... $$$$8MMMMMMMMMMMM........$$$$. |
| 42 | $$$$.. ..$$$$$$.MMMMMMMMMMM....... $$$$, |
| 43 | $$MM .M .$$$$= IMMMMMMMM$$M ...M..MM$$$7 |
| 44 | $$$$MM7M?MMM$ MMMMMMMM$$$MM...M.M~$$$$7 |
| 45 | $MMMMMMMIMMMM MMMMMMMZ$$$.MMMMMMMMMMM$$ |
| 46 | $$$$..MMMMMMMMMMMMMMMM$$$$. MMMMMMM$$$$$ |
| 47 | $$$$. .MMMMMMMMMMMMMM.$$$$. . ..$$$$7 |
| 48 | $$$$....MMMMMMMMMMMM .$$$$.. . . $$$$7 |
| 49 | $$$$.... MMMMMMMMM .$$$$$. . . . .$$$$$ |
| 50 | $$$$......MMMMMMMMM$$$$$$$$$=====7$$$$$7 |
| 51 | $$$$$ ..MODMMMMMMMM.$$$$$$$$$$$$$$$$$. |
| 52 | =$$$$........$$$MMMMMMM$$$$$$$$$$$$$$$. |
| 53 | .$$$$$7............MMMMMM . . .. . |
| 54 | ..$$$$$$........ .. ..MMM . . |
| 55 | ...$$$$$$$,........ ..MMMM . .. .. . |
| 56 | .....$$$$$$$$$$$$$$$MMMM$$$$$MM. . . . |
| 57 | .......$$$$$$$$$$$$MMMMM$$M$MM$. . ... |
| 58 | .... .+$$$$$$DMMM$M8MMMMM$$......... |
| 59 | .... . .. . .MMMMMMMMMMMMMMM. . .. |
| 60 | .... . . ...+. . ... .... |