User-agent: * Crawl-Delay: 2 Disallow: /cart/ Disallow: /training/registration/ Disallow: /training/register/ User-agent: AhrefsBot Disallow: / User-agent: CCBot Disallow: /