Who we are
ScanFree (a service of LMR Enterprises, Anza, California) checks whether AI assistants recommend a business. When someone enters a website address, our crawler reads that site, and the sites of a few competitors that the assistants named, to compare simple things like a sitemap, robots.txt rules, page titles and structured data. Contact: hello@scanfree.dev.
How to recognise us
User agent: ScanFreeBot/1.0 (+https://app.scanfree.dev/bot)
Requests come from the server at 2.24.222.13. We always send the user agent above and never pretend to be another bot or a browser.
What we do
- We read
/robots.txtfirst and follow it. If it disallowsScanFreeBot(or everyone) from a page, we do not fetch that page. - We read your sitemap and at most 12 pages of a site that was submitted, or 6 pages of a competitor's site. We wait at least one second between requests to the same site (longer if your
Crawl-delaysays so). - We fetch plain HTML only. We do not run JavaScript, submit forms, log in or follow links to other sites.
- We keep derived facts only (for example "has a sitemap" or "title is 54 characters"), and the robots.txt and response headers we received, not copies of your pages.
- We only fetch public web addresses.
How to stop us
Add this to your robots.txt:
User-agent: ScanFreeBot Disallow: /
We will stop reading your site the next time we check. You can also write to us and we will add your site to a block list.
If we blocked you by mistake, or you were blocked
If your site answers our crawler with an error or a bot-protection page, ScanFree tells the person who asked that their site (or a competitor's) could not be read. That is a finding, not an attack. Nothing is retried in a loop.