User-agent: * Crawl-delay: 10 Disallow: /ctf/ Disallow: /downloads/ Disallow: /blog/static/ # archive.org is welcome here # /ctf/ is still excluded because I use it as a C2 etc. during CTFs. There is nothing there of value to archive. User-agent: ia_archiver Disallow: /ctf/ Allow: /