How to Check if AI Can Find Your Website (Free)

To check whether AI tools can find your website, look at three things: whether your robots.txt lets AI crawlers in, whether your key content is readable in the page HTML, and whether your site describes your business in structured data. You can check most of it yourself in about 15 minutes, or run our free AI visibility check and get a score in seconds.

Is my robots.txt blocking AI crawlers?

Open yourdomain.com/robots.txt in a browser. Look for "User-agent" lines naming these crawlers, followed by "Disallow: /":

• GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI / ChatGPT)
• ClaudeBot and Claude-SearchBot (Anthropic / Claude)
• PerplexityBot (Perplexity)
• Google-Extended (Google's AI models, including Gemini)
• Bingbot (Bing, which also feeds Copilot)

A "Disallow: /" under any of those blocks that crawler. Some businesses block training bots on purpose, and that's a fair choice. But blocking search bots like OAI-SearchBot or PerplexityBot keeps you out of their answers. Also check for a blanket "User-agent: *" with "Disallow: /", which blocks everyone.

Is my site telling search engines not to index it?

View the page source of your homepage and search for noindex. If you see a robots meta tag with noindex, search engines and AI tools are being told to leave the page out. This happens more than you'd think, usually left over from a staging site.

Do I have a sitemap?

Try yourdomain.com/sitemap.xml. A sitemap lists your pages so crawlers find all of them. Most platforms, Webflow included, can generate one automatically. Adding a "Sitemap:" line to robots.txt helps too.

Can AI read my content without JavaScript?

Many AI crawlers don't run JavaScript. If your key content only appears after scripts load, they may see a nearly empty page. A quick test: view the page source and search for a sentence from your homepage. If it's not in the source, crawlers might not see it either.

Does my site have structured data?

Structured data (schema.org markup, usually JSON-LD) tells machines exactly who you are. At minimum, add Organization or LocalBusiness schema with your name, logo, address, phone and links to your profiles. Google's Rich Results Test will show what's on a page.

Does my site answer the questions customers ask?

AI tools quote clear answers. Write plain answers to the questions people actually ask before hiring you, like pricing, service area, who you serve and how long things take. An FAQ section with FAQPage schema helps.

Should I add an llms.txt file?

It's optional. llms.txt is an emerging convention: a short plain-text file at yourdomain.com/llms.txt that summarizes who you are and links your most important pages. It's not a standard yet, so do the basics above first.

Are my titles and descriptions clear?

Each page should have one clear H1, a title around 30 to 65 characters that says what you do and where, and a meta description that sums up the page in plain language. Vague titles make vague answers.

What's the fastest way to check all of this?

Run our free AI visibility check. Enter your website and you get an instant score across 16 technical checks, including AI crawler access, noindex, sitemap, readable content, structured data, FAQs, llms.txt and page basics, plus your top priorities. Add your email to see the full fix list.

The free check covers the technical signals. It doesn't test what ChatGPT or Google's AI actually say about you. That's what our full AI visibility audit does.

Quick answers

How do I know if ChatGPT can access my website?

Check yourdomain.com/robots.txt for rules that disallow GPTBot, OAI-SearchBot or ChatGPT-User. If none block them and your pages aren't set to noindex, ChatGPT's crawlers can reach your site.

What is llms.txt?

An emerging, optional plain-text file at the root of your site that summarizes your business and links your key pages for AI tools.

Is there a free AI visibility check?

Yes. The R&D Dept.'s free check scores your site across 16 technical AI visibility signals in seconds at rad.agency.

Want help fixing what the check finds? Let's talk.