Crawler readiness
In short
This finds what stops AI and search engines from reading your website. You give it a finished ScanFree scan. You get back a short list of fixes in order, each with exactly what to change and how to check it worked.
Needs and gives back
| When | Use it when AI or search engines may not be reading the site, or when a scan shows a crawler check that did not pass. |
| Needs | From your ScanFree scan or check: the crawler checks. |
| Gives back | A short list of changes in order, each with the exact file or setting to change (robots.txt lines, a meta tag, a header, a CDN setting), and how to check it worked. |
| Label | Free |
| Cost | Free. It runs on your AI account. |
Install
Claude Code and other agents that read a skills folder
mkdir -p .claude/skills/crawler-readiness && curl -fsSL -o .claude/skills/crawler-readiness/SKILL.md https://scanfree.dev/skills/crawler-readiness/SKILL.md
Claude, ChatGPT, Cursor and other MCP apps
Connect the ScanFree MCP server. This skill is already there as the prompt crawler-readiness.
First prompt to try
Run the crawler readiness skill on my latest ScanFree scan of example.com.
The whole skill
Raw file--- name: crawler-readiness description: Fix what stops AI crawlers and search engines reading your site. Use it when AI or search engines may not be reading the site, or when a scan shows a crawler check that did not pass. --- # Crawler readiness ## In short This finds what stops AI and search engines from reading your website. You give it a finished ScanFree scan. You get back a short list of fixes in order, each with exactly what to change and how to check it worked. ## When to use it Use it when AI or search engines may not be reading the site, or when a scan shows a crawler check that did not pass. ## What you need - The business's name and website. - Needed: The crawler checks: whether AI crawlers can read the site. Where to get it: a finished free scan's report (`get_scan`). ## Steps 1. If the ScanFree MCP server is connected, get its prompt `crawler-readiness` with `scan_id`, `site` or `order_id` (for a paid order, `check_id` too if the person names a check). It comes back as the task below with the data filled in: follow it and skip step 2. If it says data is missing, tell the person what it needs and stop. 2. Without the prompt, collect each item under What you need from where it says. If a needed item is missing, say what is missing and stop. 3. Do the task below with that data. ### The task You are helping the business. ScanFree checked whether AI crawlers can read this site. The results are below, each with what was found and, where something failed, the fix. Work through every line that is not a pass: 1. Say in one sentence what it means for this site. 2. Give the exact change (the robots.txt lines, the meta tag, the header or the CDN or host setting), for the platform the site runs on if you can tell. 3. Say how to check the change worked. If every line passes, say so in one sentence and stop. Do not change anything that passed. If a line says to check by hand, give the steps and say what to look for. Never suggest pretending to be a crawler or changing the user agent. **Data: the crawler checks: whether AI crawlers can read the site.** Needed. ## Rules - Counts first. Say "named in 2 of 6 answers", with both numbers. Never give a percentage on its own. - Use only facts the data or the owner gives you. Never invent prices, hours, reviews, ratings, quotes or customers. Where a fact is missing, ask the owner or leave a [check] mark. - Never promise or suggest that an AI will name the business. AI answers change from one ask to the next. - Anything you write is a draft. The owner reads it, changes it if needed, and posts or sends it themselves. Never post, send or publish for them. - If data the skill needs is missing, say what is missing instead of guessing. ## What to give back A short list of changes in order, each with the exact file or setting to change (robots.txt lines, a meta tag, a header, a CDN setting), and how to check it worked. --- ScanFree skill `crawler-readiness`, version 2. Made from the same text as ScanFree's `crawler-readiness` MCP prompt. Page: https://scanfree.dev/skills/crawler-readiness