SitejarBot

What it is

SitejarBot is the automated crawler that runs when someone requests a Sitejar compliance scan of a site - your own, or a site a Sitejar user has asked us to scan. It fetches a bounded number of pages from the target site to check for accessibility, security, privacy, deceptive-design-risk, quality, and SEO issues, following the same crawl-etiquette rules as any well-behaved bot: it identifies itself, and it honors robots.txt.

How to identify it

Every request SitejarBot makes sends this exact User-Agent header:

Opting out

If you'd rather SitejarBot not crawl your site, add this to your robots.txt:

SitejarBot checks robots.txt before crawling and will not fetch any path disallowed for its user-agent group (or the wildcard * group, if you have no SitejarBot-specific one). Rules are matched against the full path and query string, so Disallow: /search? is honored, and pattern rules using * and $ are honored too, so Disallow: /*/private and Disallow: /*.pdf$ both work as you would expect. Where two rules match, the more specific one wins, and where two are equally specific the Allow wins. If you list the same user-agent more than once, every block for it applies.

If your robots.txt is missing, or returns a 404 or any other 4xx status including 403, SitejarBot treats that as no restriction and scans normally. If it cannot be read at all - the server returns a 5xx error, the connection fails, or the request times out - SitejarBot does not guess. It scans nothing and reports why, because a file that says which pages a crawler may visit is not something to assume the contents of. That follows RFC 9309, the robots.txt standard. Starting a new scan once your server recovers will work.

SitejarBot does not currently honor the Crawl-delay directive. It fetches at most two pages at a time, and a single scan is bounded by the requesting plan to at most 100 pages, but if you have set a crawl delay, be aware it is not applied.

What it never does

SitejarBot only reads pages - it never submits forms, clicks through paywalls or logins, or attempts to access anything not linked from a normal crawl. See our Privacy Policy for what it stores about what it finds.

Contact

Questions about SitejarBot's behavior on your site can be sent to [email protected].