Docs · Site and content

Let AI crawlers in

AI assistants can only quote what their crawlers can read. aeoo checks each crawler on every crawl and, when one is blocked, gives you the exact robots.txt or setting that lets it in.

What aeoo checks

On every crawl, aeoo reads robots.txt as each crawler would, and asks the site as some of them. A crawler counts as blocked when robots.txt shuts it out, or when the server turns it away but lets other visitors in.

OAI-SearchBot

ChatGPT search

ChatGPT-User

ChatGPT browsing

PerplexityBot

Perplexity

Claude-SearchBot

Claude search

Googlebot

Google search and AI Overviews

Google-Extended

Gemini answers and training

Bingbot

Bing and Copilot

GPTBot

Training only, for OpenAI’s models

ClaudeBot

Training only, for Claude’s models

Blocking a training crawler keeps the site’s text out of future models, not out of answers, so Site health shows those on a quiet line of their own and rates the issue Low.

Can AI read your site?

Free · no sign-up

https://yourcompany.com

Run again

Checking yourcompany.com…

5 of 6answer crawlers can read yourcompany.com

OAI-SearchBotChatGPT searchWaiting

Claude-SearchBotClaude searchWaiting

PerplexityBotPerplexityWaiting

GooglebotGoogle search and AI OverviewsWaiting

Perplexity is turned away. aeoo drafts the fix and checks it again after every crawl.

What else does AI miss?Book a demo

1Open Fix now

In Progress → Site health, choose Fix now on the AI crawlers card (How to let them in, when only training crawlers are blocked) or on the AI assistants can’t read the site issue. aeoo checks right then, and opens or closes the issue to match.

2Turn off Cloudflare’s robots.txt lines, if asked

When Cloudflare adds the blocking lines, changing the file alone won’t help. In Cloudflare, open the domain, go to Security → Settings, filter by Bot traffic, open AI Crawler and Agent access policies, turn off Bot Preference Sync and save. The new file keeps Cloudflare’s other lines.

3Paste the new robots.txt

Choose Copy robots.txt. It’s the whole file: the blocked crawlers allowed at the top, and everything else as it was. Tick Let training crawlers in too if you want those in as well. Then paste it:

Webflow

Site settings → SEO → robots.txt. Paste, save, then publish the site.

WordPress

Yoast SEO → Tools → File editor, or Rank Math → General settings → Edit robots.txt. Otherwise, the file in the site’s main folder.

Shopify

Online Store → Themes → Edit code. Add the robots.txt template and replace what’s in it.

HubSpot

Settings → Content → Pages → SEO & Crawlers. Pick the domain, edit robots.txt and save.

Anything else

Replace the robots.txt file wherever the site is hosted.

4Let them past the firewall, if asked

If the server turns a crawler away, the fix is a setting, not a file. On Cloudflare, set Search and Agent to Allow in AI Crawler and Agent access policies, and check that Bot Fight Mode is off. Elsewhere, allow the crawlers at the host, the CDN or a security plugin such as Wordfence.

5Check again

Choose Check again. New settings can take a few minutes to show.

If the site turns aeoo away

aeoo’s own crawler is AEOAuditBot. If a firewall turns it away, Site health says The site turned our crawler away. Allow AEOAuditBot in the firewall, Cloudflare (a WAF custom rule that skips it) or security plugin, keep robots.txt open to it, then choose Crawl now. The card links to the crawler’s public page.