How to check AI crawler access

Start with one page and one question

Choose a page that answers a real customer question. Record its canonical URL, the question, and the answer you expect a reader to find. Crawl access matters only if the page itself is useful and visible. A successful fetch is a prerequisite for some search systems, not proof that an assistant will cite the page.

Check the page response

Open the URL without signing in. Confirm it returns a successful response, renders the key answer as HTML text, and does not redirect to an unrelated page. Check the canonical link and any noindex or snippet restriction that might prevent a search engine from showing the content. Repeat on the mobile version if the site serves different markup.

For Google AI Overviews and AI Mode, Google says a supporting page must be indexed and eligible to appear in Search with a snippet . It says there is no separate AEO markup requirement, and that eligibility still does not guarantee crawling, indexing or serving.

Read the robots rules that apply

Fetch the site's robots.txt and read the group matching the crawler you care about. A blanket block on the page or its path may prevent search retrieval. Check the page URL itself as well as shared paths for assets needed to understand the answer. Do not assume that a line for one bot controls every AI product.

OpenAI documents OAI-SearchBot for ChatGPT search separately from GPTBot for model training . Their controls are independent. If ChatGPT search visibility is the goal, inspect the OAI-SearchBot rule and relevant server access. An allow rule is a crawl policy, not a citation guarantee.

Google says Googlebot is the crawl control for its Search AI features . Google-Extended addresses other uses; it is not a replacement for checking Googlebot and ordinary Search eligibility.

Check the layer in front of the site

A permissive robots.txt does not prove the crawler reaches the page. Review CDN, firewall, bot-management, login and geographic rules. Look for blocked requests on the page and robots.txt paths, record the status code and time, and compare with the crawler's documented identity. Avoid solving a block by opening private routes or removing protections that the public page does not need.

Google's guidance explicitly mentions CDN and hosting infrastructure alongside robots.txt . OpenAI also publishes crawler IP ranges in its bot documentation; use those records when investigating requests, while keeping your normal security checks.

Verify indexing and repeat the observation

For Google, inspect the URL in Search Console and use the reported crawl, indexing and rendered-page details to narrow the issue. After fixing a block, allow time for recrawling; Google says it can take days to months.

Then run a small, dated prompt sample. Save the exact prompt, engine, date, answer and cited URLs. Repeat the same sample later. A missing citation can reflect many causes beyond access, including source selection and answer variation, so do not attribute a change to a robots edit without better evidence.

A compact checklist

Record: canonical URL; page response and visible answer; indexing and snippet status; applicable robots groups; CDN or firewall result; timestamp of any change; and the repeated prompt observations. Keep this with the page so the next review can tell what changed.

Next, use the answer-ready page guide for editorial quality and the citation measurement guide for the prompt log. Sources were checked on 28 September 2026.