# Stolen from https://git.sr.ht/robots.txt # Too aggressive, marketing/SEO User-agent: SemrushBot Disallow: / # Too aggressive, marketing/SEO User-agent: SemrushBot-SA Disallow: / # Marketing/SEO User-agent: AhrefsBot Disallow: / # Marketing/SEO User-agent: dotbot Disallow: / # Marketing/SEO User-agent: rogerbot Disallow: / User-agent: BLEXBot Disallow: / # Huwei something or another, badly behaved #User-agent: AspiegelBot #Disallow: / # Marketing/SEO User-agent: ZoominfoBot Disallow: / # YandexBot is a dickhead, too aggressive #User-agent: Yandex #Disallow: / # Marketing/SEO User-agent: MJ12bot Disallow: / # Marketing/SEO User-agent: DataForSeoBot Disallow: / # Used for Alexa, I guess, who cares User-agent: Amazonbot Disallow: /