Skip to content
Serpify
Free tools AI visibility check AI Overview checker Pricing
Sign in Start free trial

Free tool

Free AI Crawler Checker

Check whether GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and 15 more AI crawlers are allowed by your robots.txt, and get the exact lines to fix it.

Run the free check

See whether ChatGPT, Claude, Perplexity, Gemini and 16 other AI crawlers can read your site.

Free, no account needed.

What is an AI crawler checker?

An AI crawler checker reads your robots.txt and tells you which AI bots may crawl your site. It separates the crawlers that decide whether you can be found and cited in AI answers, such as OAI-SearchBot and PerplexityBot, from the ones that only collect training data, such as GPTBot and CCBot, so you can block one without losing the other.

How it works

  1. 1

    Enter your domain

    No signup. We fetch your robots.txt the way crawlers do.

  2. 2

    We apply the rules

    Each crawler follows its own group, else the "*" group; the most specific rule wins, as Google documents.

  3. 3

    Copy the fix

    Get ready-to-paste robots.txt lines for any AI search crawler you block by mistake.

What you get

AI search and answer crawlers

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot and more.

User-triggered fetchers

ChatGPT-User, Claude-User, Perplexity-User and others that open pages when someone asks.

Training crawlers

GPTBot, ClaudeBot, Google-Extended, CCBot and others you may choose to block.

Site-wide AI readiness Paid plan

Crawler access is one part; paid plans track whether AI engines actually mention and cite you.

What the paid plan adds

The free check shows one page, one moment. Serpify runs these checks for every client, every day, and turns them into branded reports and proposals.

Start your 14-day free trial

No card needed.

FeatureFreePaid
robots.txt check for 20 AI crawlers Included Included
Copy-paste fix Included Included
AEO & GEO grading of every client page — Included
Daily AI mention and citation tracking — Included

Who it’s for

Agencies

Catch a client’s CDN or robots.txt blocking AI search before it costs them citations.

Freelancers

A quick, concrete win to open an AI search engagement.

In-house teams

Decide on training crawlers without accidentally blocking AI search.

Why it matters

Separate bots

OpenAI uses OAI-SearchBot for ChatGPT search results and GPTBot for training; blocking GPTBot does not remove a site from ChatGPT search.
Source: OpenAI, Overview of OpenAI crawlers

Google-Extended

Google-Extended controls use for Gemini training and grounding; it does not affect inclusion in Google Search.
Source: Google, Google’s common crawlers

How we check

We fetch /robots.txt over HTTPS and apply Google’s documented matching rules: a crawler follows the group naming its token, otherwise "*"; the longest matching rule wins and Allow wins a tie. A missing robots.txt allows everything; a server error is treated as blocking everything.

Built and maintained by the Serpify team. Updated . Serpify is a rank tracker and AI search visibility platform for SEO agencies and freelancers. It tracks Google rankings to position 100, AI Overview citations, and brand mentions in ChatGPT, Gemini and Google AI Mode, with white-label client reports.

Questions

More free checks on the free tools page.

Will blocking GPTBot remove me from ChatGPT?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and pages opened for a user use ChatGPT-User. Keep those allowed to be found and cited.

I have no robots.txt. Is that a problem?

No. Without a robots.txt every crawler is allowed. Adding one is still useful to point crawlers to your sitemap.

Does a firewall or CDN affect this?

Yes. Some CDN bot settings block AI crawlers even when robots.txt allows them. This tool checks robots.txt; check your CDN’s bot rules too.

How often should I check?

After any site migration, CMS update or change to your CDN settings.

Related free tools

AEO & GEO Grader

Score any page 0-100 on how ready it is to be found and cited by ChatGPT, Gemini, Perplexity and AI Overviews.

Open the tool

llms.txt Validator & Generator

Check your llms.txt against the standard, or get a ready-to-upload draft built from your sitemap.

Open the tool

Website SEO Checker

Audit any page for on-page SEO: titles, headings, links, images, speed and indexability, with a fix for every issue.

Open the tool

Want this tracked every day, for every client?

Start a 14-day free trial with the full product. No card needed; your tracking history stays when you pick a plan.

Start free trial See pricing