Crawler
SeoAutopilotBot, the crawler behind ShownTrue
If this address is in your server logs, someone using ShownTrue added your site and started a crawl of it, or connected your WordPress site to publish to and the product checked a page on it. Neither happens unless a person starts it. This page says what the crawler does and how to stop it.
How it identifies itself
Every request carries this user agent:
SeoAutopilotBot/0.1 (+https://showntrue.com/bot; respects robots.txt)
Pages that only build themselves with JavaScript are fetched a second time in a real browser, which sends the same user agent.
What it obeys
- It reads your robots.txt before any other page and follows the group for SeoAutopilotBot if there is one, otherwise the group for *. Only one group applies, so a SeoAutopilotBot group replaces your * rules for this crawler rather than adding to them.
- If your robots.txt times out, fails with a server error, or answers 401 or 403, it fetches nothing else from the site. A robots.txt that answers 404 means no rules, as the standard says.
- It refuses any address on a private or internal network, so it cannot be pointed at a server behind your firewall.
How hard it works your server
Unless the person running the crawl changed them, these are its limits:
- Requests per second
- 2
- Pages per crawl
- 500
- Links deep from the start page
- 5
- Longest a crawl may run
- 30 minutes
- Longest it waits for one page
- 20 seconds
It does not read Crawl-delay yet. The pace above applies whatever your robots.txt asks for, so if you need it slower, block it as below and write to us.
How to block it
Add this to your robots.txt:
User-agent: SeoAutopilotBot Disallow: /
The next crawl reads it first and stops there. To check what a rule does before you publish it, paste your file into the robots.txt tester with SeoAutopilotBot as the user agent.
Anything else, write to support@showntrue.com with your site’s address.