Great SEO Tool crawlers
Great SEO Tool is a technical SEO audit tool. Its crawlers run only when a person asks us to audit a site, so you won't see them crawling the web at random. Each one has its own name, so you can allow or block each job separately.
All of them follow your robots.txt. Each bot looks for a group that matches its own name first and falls back to the * group, and each honours a Crawl-delay if you set one.
SiteAuditBot
User-agent token: GreatSEOTool-SiteAuditBot
Crawls the pages of a site to check titles, meta descriptions, headings, links, redirects, canonical tags and other technical SEO details. This is the main audit crawler.
LinkCheckBot
User-agent token: GreatSEOTool-LinkCheckBot
Checks whether external links found on an audited site still load. If you see it in your logs, someone's site links to yours and an audit confirmed the link still works.
ReachabilityBot
User-agent token: GreatSEOTool-ReachabilityBot
Makes one lightweight request to confirm an address resolves before an audit starts.
AssetCheckBot
User-agent token: GreatSEOTool-AssetCheckBot
Checks that images, JavaScript and CSS files load, and reads their headers for compression and caching. It's separate because it requests every static file, not just pages.
RenderBot
User-agent token: GreatSEOTool-RenderBot
Loads a page in a real headless browser when a page appears to depend on JavaScript, and for homepage screenshots. This is a heavier request pattern, so you can control it separately.
How do I block or slow a crawler?
Add a group for the crawler's token to your robots.txt file. For example, to block the browser-rendering bot and slow the audit bot:
User-agent: GreatSEOTool-RenderBot
Disallow: /
User-agent: GreatSEOTool-SiteAuditBot
Crawl-delay: 5Want to run an audit yourself? Audit your own site for free, or read more about the tool on the About page.