Launch ToolkitCalculatorsConvertersDeveloperTextAll toolsLaunches
Launch your project Sign in
Theme
Home/Answers/Launch
How-to

How do I know if ChatGPT and other AI assistants can see my website?

Short answer

Check three things: your robots.txt must not block the AI search crawlers (OAI-SearchBot for ChatGPT search, PerplexityBot, Claude-SearchBot, plus Googlebot and Bingbot), your firewall must not refuse them, and your page text must be in the HTML rather than only appearing after JavaScript runs. An AI search visibility checker tests all three for one page.

Search crawlers versus training crawlers

AI companies run separate crawlers. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for model training, for example. You can block training crawlers and still allow the search ones, so you can be cited without contributing to training data.

Blocks you do not see in robots.txt

Bot protection and firewall rules can refuse AI crawlers even when robots.txt allows them. Single page apps can also serve almost empty HTML, which many crawlers read without running scripts. Both make you invisible to assistants that would otherwise quote you.

Step by step

  1. Run the check. Paste your page into an AI search visibility checker.
  2. Unblock search crawlers. Allow OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot and Bingbot in robots.txt.
  3. Serve real HTML. Make sure your main text is in the server HTML, and add structured data.

Frequently asked questions

Should I block GPTBot?

It is your choice. GPTBot is used for training; blocking it does not stop ChatGPT search, which uses OAI-SearchBot.

Is llms.txt required?

No. Some AI tools read it, but Google has said it does not use it.

How long until AI assistants pick up changes?

It depends on when their crawlers next visit. There is no way to force it, but a sitemap and good internal links help.

Related answers