How to Block AI Crawlers Without Hurting SEO in 2026

If you run a blog, publishing site, SaaS website, news platform, or knowledge base in 2026, you may eventually ask whether you should block AI crawlers from reading your content. It sounds like a simple decision. You create the article, an AI company crawls it, and you stop the crawler. But the situation has become more complicated because the same AI company may operate one bot for model training, another for search, and another that visits a page because a real user asked an AI assistant to access it.

That difference matters more than many website owners realize. You may want to prevent your original research or premium content from being collected for model training while still wanting your articles to appear in ChatGPT, Perplexity, Google, Siri, and other discovery experiences. If you simply block every bot with “AI” in its name, you could protect one part of your content strategy while damaging another.

This guide explains how AI crawling actually works in 2026, which bots serve different purposes, how robots.txt fits into the picture, where Cloudflare AI Crawl Control can help, and how publishers can make a smarter decision between allowing, blocking, and even charging AI crawlers.

Why Website Owners Want to Block AI Crawlers in 2026

For years, the relationship between websites and search engines was relatively easy to understand. Search engines crawled pages, indexed them, displayed links, and potentially sent visitors back to the original website.

Generative AI has changed that relationship.

An AI system may access information from a website and use it to support an answer directly inside an AI interface. Sometimes that answer includes a citation or link. Sometimes the user gets enough information from the answer that they never visit the source.

For a hobby website, that may not feel important. For a business built on original reporting, tutorials, paid research, comparison content, reviews, or proprietary information, the question becomes much more serious.

Should an AI company be allowed to collect the content?

Does that crawler send meaningful referral traffic?

Is the content being accessed for search, model training, or a user requested action?

Can the publisher allow one use while refusing another?

These questions are one reason Cloudflare has expanded its AI crawler controls so aggressively. Its current AI Crawl Control product, previously called AI Audit, gives website owners visibility into crawler activity, individual crawler controls, robots.txt monitoring, and experimental monetization through Pay Per Crawl.

Cloudflare AI Crawl Control documentation

AI Crawlers Are Not All Doing the Same Job

Before you block AI crawlers, the first thing I would do is understand why they are visiting your site.

Cloudflare now classifies AI related traffic according to behavior rather than putting everything into one generic AI bot category. Its current model separates Search, Agent, and Training activity. Search crawlers collect or index information for later discovery. Agent traffic acts on behalf of a user in real time. Training crawlers collect information for model training or fine tuning.

That classification is much more useful than thinking about every AI crawler as the same thing.

Consider OpenAI.

Cloudflare currently identifies GPTBot as an AI crawler, OAI SearchBot as AI Search, and ChatGPT User as an AI Assistant. Anthropic similarly operates ClaudeBot, Claude SearchBot, and Claude User. Perplexity operates PerplexityBot for search and Perplexity User for user requested access.

Here is the practical difference.

Type	What it generally does	Typical publisher decision
Training crawler	Collects content for model training or fine tuning	Consider blocking
AI search crawler	Helps surface pages in AI search results	Often allow
User initiated agent	Fetches content because a user requested it	Evaluate separately
Traditional search crawler	Builds normal search indexes	Usually allow
Unknown scraper	Collects content without clear identity or value	Usually investigate or block

This is why a blanket rule can create unexpected consequences.

[Add image above this section: Training Crawlers vs AI Search vs User Agents with Allow, Block, or Charge paths]

Blocking the Wrong Bot Can Hurt Your Visibility

The most important SEO lesson in this entire article is simple: blocking AI training and blocking AI discovery are not necessarily the same action.

OpenAI makes this distinction very clearly.

Its current publisher guidance says websites that want their content included in summaries and snippets in ChatGPT search should make sure they are not blocking OAI SearchBot. OpenAI separately tells publishers who want to exclude pages from potential training to disallow GPTBot.

That means a publisher could potentially block GPTBot while still allowing OAI SearchBot.

For me, this is a much better approach than seeing “OpenAI” in a crawler list and immediately blocking everything.

The same principle appears in Google’s and Apple’s crawler controls.

GPTBot vs OAI SearchBot

If you publish content and care about visibility inside ChatGPT, this distinction deserves its own section.

GPTBot is associated with content that may be used to help improve OpenAI models. OAI SearchBot, by contrast, helps content appear in ChatGPT search experiences.

OpenAI also says publishers who allow OAI SearchBot can track referral visits from ChatGPT because referral URLs include a utm_source=chatgpt.com parameter.

That gives publishers a much more useful strategy than blindly blocking.

You can ask:

How much is ChatGPT crawling?

How much referral traffic is ChatGPT sending?

Which pages receive that traffic?

Does allowing search discovery create enough value to justify continued access?

That moves the conversation from fear to measurement.

Google Extended Lets You Separate Search From Certain AI Uses

Google provides another useful example.

Google Extended is a special robots.txt control that publishers can use to manage whether content crawled from their sites may be used for training future generations of Gemini models and certain Gemini grounding uses.

The critical SEO point is that Google states Google Extended does not affect inclusion in Google Search and is not used as a Google Search ranking signal.

So if your goal is to limit certain Gemini related uses, do not casually block Googlebot.

Googlebot is what Google Search uses to discover and index web content. Google Extended provides a more specific control.

A robots.txt rule can look like this:

User-agent: Google-Extended
Disallow: /

That is very different from:

User-agent: Googlebot
Disallow: /

The second rule can have serious implications for normal search visibility.

Google Extended crawler documentation

Applebot Extended Works in a Similar Way

Apple has also separated normal discovery from model training control.

Applebot is used across experiences such as Spotlight, Siri, and Safari. Apple says website owners who allow Applebot can have their content appear in those discovery experiences.

Applebot Extended gives publishers an additional control over whether content crawled by Applebot can be used to train Apple’s general purpose foundation models.

Most importantly, Apple says pages that disallow Applebot Extended can still remain available to Applebot for search and discovery.

A basic rule looks like this:

User-agent: Applebot-Extended
Disallow: /

Again, the lesson is not “block Apple.”

The lesson is to understand which control affects which use.

What About PerplexityBot?

Perplexity is slightly different because its public crawler documentation identifies two important user agents.

PerplexityBot is designed to surface and link websites in Perplexity search results. Perplexity says it is not used to crawl content for foundation model training.

Perplexity User supports actions initiated by a person using Perplexity. When someone asks a question, the service may visit a webpage to help answer it and provide a link.

Perplexity also states that because Perplexity User fetches are initiated by a user, this fetcher generally ignores robots.txt.

That last part is worth paying attention to.

It demonstrates why robots.txt is useful, but not a technical security boundary.

Perplexity crawler documentation

How to Block AI Crawlers With robots.txt

The simplest way to block AI crawlers that respect publisher directives is through the robots.txt file located at the root of your domain.

For example, if you want to prevent GPTBot from crawling your entire website, you can use:

User-agent: GPTBot
Disallow: /

If you wanted to prevent several selected training related crawlers while leaving normal search crawlers available, the file could contain separate groups.

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

This is deliberately different from blocking every crawler with:

User-agent: *
Disallow: /

A wildcard rule like that can create much wider crawling problems.

Also remember that robots.txt controls crawling, not necessarily indexing in every scenario. Google explicitly warns that robots.txt should not be treated as the mechanism for hiding a webpage from Google Search. A URL can sometimes still appear if Google discovers it through external links. If your goal is to remove a page from search results, controls such as noindex, authentication, or removing public access may be more appropriate.

Why robots.txt Alone Is Not Enough

This is where many AI crawler tutorials become misleading.

robots.txt is not a firewall.

Cloudflare’s current documentation describes robots.txt compliance as voluntary. The file communicates what you want a crawler to do, but it does not technically prevent an uncooperative bot from requesting your pages.

I think of it like a sign on a private office door.

A respectful visitor reads the sign and stays out.

The sign itself does not physically lock the door.

When a crawler ignores your directives, stronger controls may be necessary. That can include a web application firewall, bot management, request verification, authentication, or a dedicated crawler control service.

Cloudflare AI Crawl Control Explained

For websites already using Cloudflare, AI Crawl Control is becoming one of the most practical ways to manage this problem without manually maintaining an enormous crawler list.

The product was originally introduced as AI Audit, but Cloudflare now calls it AI Crawl Control.

The current dashboard can show which AI services are accessing your website, reveal request patterns, allow granular crawler policies, track robots.txt compliance, and connect to Pay Per Crawl functionality. Cloudflare says AI Crawl Control is available across its plans.

What I like about this approach is visibility.

Before blocking anything, you can see who is actually crawling.

That matters because many website owners have strong opinions about AI crawlers without knowing how frequently those crawlers visit their site or whether those same companies are sending referral traffic back.

A good crawler policy should begin with data.

Cloudflare Is Changing AI Bot Controls in September 2026

There is another reason this topic matters right now.

Cloudflare says that beginning September 15, 2026, new domains will receive updated default AI bot policies.

Under the announced defaults, bots classified as Training or Agent will be blocked on pages displaying ads, while Search traffic remains allowed. Cloudflare is also moving away from its older single “Block AI bots” setting toward separate behavior based controls.

That tells us where the industry is moving.

The future is probably not:

Allow AI: Yes or No.

It is more likely:

Allow AI Search: Yes.

Allow model training: Maybe not.

Allow agents: Depends on the page.

Allow commercial crawling: Possibly, under specific terms.

This is a much more mature framework.

Can You Charge AI Crawlers Instead of Blocking Them?

This may become one of the most interesting developments for publishers.

Cloudflare’s Pay Per Crawl feature allows participating website owners to choose between Allow, Block, and Charge for supported AI crawler access.

The feature is currently in closed beta. Cloudflare says publishers can set a price for successful content retrieval, and the current minimum documented price is $0.001 per successful crawl.

Cloudflare has also added dynamic pricing capabilities.

Website owners can potentially vary pricing according to content or request conditions through response headers or Cloudflare Workers. That means a business could eventually treat ordinary public content differently from highly valuable reports, premium archives, or specialist data.

I would not describe this as a proven new revenue stream yet because Pay Per Crawl is still a beta product and the economics are developing.

But the direction is significant.

The conversation is moving from:

“Please do not crawl my content.”

toward:

“If you want automated commercial access to this content, here are the conditions.”

Cloudflare Pay Per Crawl documentation

Allow, Block, or Charge: A Better Decision Framework

I would not recommend the same crawler policy for every website.

A technology blog that wants maximum visibility inside AI search has different incentives from a premium market research company that sells access to proprietary reports.

A useful starting framework looks like this:

Content or crawler situation	My starting approach
Google Search crawler	Allow
AI search crawler that can send citations	Usually allow and measure
Dedicated training crawler	Decide according to content strategy
Premium subscriber content	Protect more aggressively
Internal documentation	Do not expose publicly
Unknown crawler ignoring directives	Investigate and consider enforcement
Commercial AI crawler willing to pay	Evaluate charging
Sensitive customer data	Never rely on robots.txt as protection

This is why I would avoid copying a random “block every AI bot” robots.txt file from the internet.

Your crawler policy should reflect how your website makes money.

A Practical Content Protection Strategy

If I were reviewing a publisher’s website today, I would approach it in this order:

1. Identify which AI and search crawlers are actually accessing the site through server logs, analytics, CDN data, or Cloudflare AI Crawl Control.
2. Separate normal search, AI search, user initiated agents, and training activity instead of treating them as one category.
3. Keep crawlers that provide useful search visibility unless you have a clear reason to restrict them.
4. Use specific robots.txt rules for training related crawlers you do not want accessing content.
5. Use infrastructure enforcement such as a WAF or AI Crawl Control when robots.txt is not sufficient.
6. Measure referral traffic, crawler volume, bandwidth, and business value before changing the policy again.

This gives you something far better than a static crawler blacklist.

It gives you a content access strategy.

When Should You Block AI Crawlers Completely?

I would block AI crawlers completely when there is no realistic business reason for that crawler to access the content, particularly when the material is proprietary, licensed, commercially sensitive, or not intended for public automated retrieval.

However, if a page is genuinely sensitive, blocking a crawler should not be your primary security control.

Do not publish customer records, confidential business documents, private dashboards, personal data, or internal files publicly and assume a robots.txt instruction protects them.

Use authentication and proper access controls.

robots.txt is about crawler preferences.

It is not a replacement for website security.

AI Agents Make This Problem Even More Complicated

There is another reason website owners should start thinking beyond traditional scrapers.

AI agents increasingly browse websites in real time.

Instead of crawling millions of pages to build an index, an agent may visit one specific page because a user asked it to check a price, research a product, download information, or complete an online task.

This is part of the same shift I explored in our guide to AI browser agents, where tools are moving from simply reading websites toward actually interacting with them.

Read our AI Browser Agents guide

OpenClaw provides another example of this agent based future. These systems can connect reasoning models with browsers, files, applications, and other tools. As agents become more capable, website owners will increasingly need policies that distinguish automated research, search indexing, model training, and real user initiated actions.

Read our OpenClaw AI Agent guide

That is why Cloudflare’s new Search, Agent, and Training classification is worth watching closely.

Does Blocking AI Crawlers Hurt SEO?

The answer depends entirely on which crawler you block.

Blocking GPTBot does not mean the same thing as blocking OAI SearchBot.

Blocking Google Extended does not mean the same thing as blocking Googlebot.

Blocking Applebot Extended does not necessarily stop Applebot from supporting search and discovery.

This distinction is the heart of modern crawler management.

Cloudflare also warns publishers using Pay Per Crawl that blocking or charging search engine crawlers may negatively affect SEO because those crawlers may no longer be able to index content properly.

So never make crawler decisions based only on the company name.

Make them based on crawler purpose.

Frequently Asked Questions

How do I block AI crawlers without hurting SEO?

Start by identifying the exact crawler and its purpose. Keep traditional search and AI search crawlers available when they provide useful discovery, while selectively restricting training specific crawlers such as GPTBot, Google Extended, or Applebot Extended according to your content policy. Always verify the current crawler documentation before changing robots.txt.

Does blocking GPTBot stop my website appearing in ChatGPT search?

OpenAI separates GPTBot from OAI SearchBot. Its publisher documentation says websites that want content surfaced and cited in ChatGPT search should allow OAI SearchBot, while GPTBot can be disallowed for pages publishers wish to exclude from potential training.

Does blocking Google Extended hurt Google rankings?

Google says Google Extended does not affect inclusion in Google Search and is not used as a ranking signal. It provides control over specific Gemini related uses of content.

Can I stop Apple from training on my website but remain in Apple search?

Apple says publishers can disallow Applebot Extended to opt out of foundation model training while still allowing Applebot to access content for experiences including Spotlight, Siri, and Safari.

Is robots.txt enough to stop an AI scraper?

No. Cloudflare describes robots.txt as voluntary. Crawlers that ignore it can still technically request public content. For stronger enforcement, use server controls, authentication, a WAF, or a crawler management service.

What happened to Cloudflare AI Audit?

Cloudflare AI Audit is now called AI Crawl Control. The product provides crawler analytics, blocking controls, directive monitoring, and access to Pay Per Crawl features.

Can websites charge AI companies to crawl content?

Cloudflare is experimenting with exactly this through Pay Per Crawl. The feature remains in closed beta, but participating publishers can set crawler actions to Allow, Block, or Charge and establish a price for successful content access.

Final Thoughts

The instinct to protect original content makes sense. Writers, publishers, agencies, researchers, and brands spend real time and money creating information that gives their websites value.

But I do not think the right answer in 2026 is to block everything associated with artificial intelligence.

The web is moving toward a more complicated model where some bots train models, some power search, and others act directly for users. A crawler that creates no value for your business should not automatically receive the same access as one that helps customers discover you.

That is why I would begin with visibility before restriction.

Find out who is crawling your site. Understand what each crawler does. Measure the referral traffic you receive. Separate search from training. Use robots.txt to express your preferences, and use proper technical controls when you genuinely need enforcement.

If you decide to block AI crawlers, do it selectively. Protect the content that creates your competitive advantage, but do not accidentally close the door on the new search and discovery channels where your future audience may already be looking for answers.

At Simplify AI Tools, we spend a lot of time looking at how new AI Tools change normal business workflows. AI crawler control is one of those topics that may look technical on the surface, but underneath it sits a much bigger question: in an AI driven internet, who gets access to the content your business creates, under what conditions, and what value comes back to you?