iLuvSEO

Article / Control access by purpose

Which AI Crawlers Should You Allow?

Search, training, and user-requested retrieval are different jobs. This guide maps the major AI crawlers to the controls and checks that actually apply.

Author
Published
Last checked
Research month
Reading time
12 minutes

The phrase “block AI bots” sounds decisive until you look at what the bots do. Search indexing, answer grounding, model training, and a live fetch requested by a user are different jobs. A serious policy has to decide which business outcome the site wants, then map the exact crawler, page control, edge rule, and verification method to that decision.

The decisions to make before editing robots.txt

  • Decide by crawler purpose instead of using one blanket AI rule.
  • A robots.txt directive is only policy. Confirm the real response and edge behavior.
  • Blocking training does not automatically remove a page from AI search or user-requested retrieval.

Crawler purpose map

Separate search, training, and user-requested access

The platform controls are not symmetrical. The safest policy starts with purpose and verifies the current first-party documentation for each token.

Separate search, training, and user-requested access
PurposeTypical control surfaceWhat the control does not do
Search indexing and AI search featuresStandard search crawler access plus index and snippet controls on the page.A guarantee of indexing, retrieval, citation, or ranking.
Separate model-training useA platform-specific training token when the platform documents one, such as GPTBot, ClaudeBot, or Google-Extended.Removal from previous collections or identical control over search products.
User-requested live retrievalThe platform's user-fetch documentation, WAF policy, authentication, rate limits, and the permissions of the requested action.That robots.txt will be honored in the same way as an automatic crawler.
Content-level preview controlnoindex, nosnippet, max-snippet, or supported data-nosnippet markup on the actual response.Control over every external copy, previous crawl, training use, or unsupported platform.
Bot verification at the edgeOfficial IP data or reverse and forward DNS checks where documented, combined with the expected user agent and behavior.A permanent identity guarantee when ranges, tokens, products, and providers can change.

01

Start with the business outcome

Before writing a directive, I ask three questions. Do we want the content eligible for search and AI answer discovery? Do we permit separate model-training use? Do we permit a live fetch that a user explicitly requested? A publisher, store, private support portal, and public documentation site can reasonably answer those questions differently.

The policy can also differ by content class. Public product facts may need discovery. Licensed research may require stricter reuse controls. Account pages should stay behind authentication regardless of the bot name. The rule should follow the content and risk, not the current mood around AI.

  • List the content classes and the owner of each decision.
  • Name the desired search, training, and user-fetch outcome separately.
  • Record the legal, contractual, privacy, security, and commercial reason for the rule.

02

The platforms do not use one shared control model

Google says Googlebot governs Search, including its AI search features. Google-Extended is a control token for specified Gemini training and grounding uses, not a separate crawler identity, and Google says it does not affect Search inclusion or act as a Search ranking signal.

OpenAI separates OAI-SearchBot for ChatGPT Search from GPTBot for possible foundation-model training. Anthropic documents Claude-SearchBot, ClaudeBot, and Claude-User as separate purposes. Perplexity documents PerplexityBot for search visibility and Perplexity-User for user-requested fetches. Those labels are similar enough to confuse and different enough that copying a random robots block is risky.

  • Use the exact, case-sensitive token from the current official source.
  • Do not assume a training rule controls search or a search rule controls user fetches.
  • Recheck every subdomain because robots.txt policy is host-specific.

03

robots.txt is a policy file, not proof of the outcome

A correct directive can still fail operationally. The CDN may serve a different robots.txt, the WAF may block an allowed crawler, a redirect may change hosts, or the crawler may receive a challenge page instead of the content. I test the public response from the real hostname and inspect the edge decision.

I also remember what robots.txt cannot do. It controls crawling behavior for compliant automatic crawlers. It does not guarantee removal from an index, erase previous collections, or force a user-requested fetcher to behave like a scheduled bot. Blocking a crawler can also prevent it from seeing a page-level noindex directive.

  • Fetch robots.txt through the public hostname and record its status, headers, and body.
  • Test one allowed and one blocked canary route for each crawler class.
  • Confirm important CSS, JavaScript, images, and API responses are not accidentally blocked.

04

Use page-level controls when the problem is the preview

Sometimes the goal is not to block discovery. It is to keep a sensitive passage out of a preview or generated answer. Google documents normal Search controls such as noindex, nosnippet, data-nosnippet, and max-snippet for its AI search features. Bing supports data-nosnippet for excluding selected sections from snippets and AI-generated answers while leaving the rest of the page discoverable.

These controls have tradeoffs. Removing a useful passage from previews can reduce how much context the system has. A crawler also needs access to read a page-level directive. I use the narrowest supported control and test the rendered result instead of assuming a meta tag has universal meaning.

  • Choose route-level blocking only when the entire route should be unavailable to that crawler.
  • Choose noindex when the page itself should not appear in the supported search index.
  • Choose snippet controls when discovery is acceptable but a specific preview is not.

05

Do not trust a user-agent string by itself

Any requester can send a famous bot name. Bing and Google explicitly warn about spoofing, and several platforms publish IP data or a verification method. At the WAF, I combine the expected user agent with official network verification when the platform provides it.

That check also needs maintenance. IP ranges, crawler versions, and infrastructure can change. A copied allowlist that never refreshes becomes either a security hole or an accidental block. The policy register should name the official source and review date beside every exception.

  • Use official IP JSON, CIDR ranges, or documented DNS verification.
  • Apply rate limits and behavioral monitoring even to verified bots.
  • Never bypass authentication or application authorization because a crawler identity looks familiar.

06

Treat crawler governance like an operating policy

I keep a register with one row per crawler and columns for owner, purpose, allowed paths, blocked paths, robots rule, WAF rule, identity source, review date, and rollback plan. That turns an invisible text file into something the business can actually govern.

After a change, I watch verified crawler requests for at least seven days and record route, status, bytes, response time, robots decision, and WAF decision. I refresh official identity data automatically where possible, review documentation monthly, and revisit the business policy quarterly.

  • Version every robots and WAF change with an owner and reason.
  • Monitor 2xx, 3xx, 4xx, 5xx, challenge, and timeout behavior separately.
  • Keep a tested rollback and confirm it from the public edge after use.

Method

How I audit an AI crawler policy

The audit checks the written intent, the public files, the page controls, the edge enforcement, and the logs. A policy is not complete until all five agree.

  1. Build the purpose register

    List every relevant bot, owner, business purpose, content class, desired access, official source URL, and review date.

  2. Read the live controls

    Fetch robots.txt, response headers, meta robots, snippet markup, authentication behavior, and canonical state from the real hostname.

  3. Inspect the edge layer

    Review CDN, WAF, rate limit, bot product, redirect, and challenge rules that can override the written crawler policy.

  4. Test canary routes

    Use one intentionally allowed and one intentionally blocked route per class, including required assets and rendered content.

  5. Verify identity and logs

    Match official network verification where available and inspect real requests, statuses, bytes, latency, and enforcement decisions.

  6. Monitor and review

    Watch at least seven days after a change, refresh identity data, keep rollback proof, and revisit platform documentation monthly.

Limits

What crawler controls cannot guarantee

robots.txt is voluntary for compliant crawlers and does not erase data already collected. Platform-specific controls can change, and user-requested fetchers may not follow the same rules as automatic crawlers.

An allow rule does not guarantee discovery, indexing, answer inclusion, or traffic. A block can also have wider consequences when a search product and an AI feature share the same crawler.

  • Blocking crawl does not necessarily remove a known URL from an existing index.
  • A crawler that cannot fetch the page may never see its noindex directive.
  • Google-Extended does not control Google Search AI features.
  • Microsoft does not present the same simple search-versus-training crawler split as OpenAI or Anthropic.
  • CDN and WAF products can change behavior outside the robots.txt file.

Sources

Primary sources I checked

  1. Google common crawlersGoogle Search Central, reviewed Opens in a new tab
  2. AI features and your websiteGoogle Search Central, reviewed Opens in a new tab
  3. OpenAI botsOpenAI Developers, reviewed Opens in a new tab
  4. How Anthropic crawls the web and how site owners can block its crawlersClaude Help Center, reviewed Opens in a new tab
  5. Perplexity crawlersPerplexity Documentation, reviewed Opens in a new tab
  6. Bing crawler verificationBing Webmaster Tools, reviewed Opens in a new tab
  7. Bing introduces support for the data-nosnippet HTML attributeBing Webmaster Blog, reviewed Opens in a new tab

Updates

What changed

  1. Published the May research article after verifying the current Google, OpenAI, Anthropic, Perplexity, and Bing control surfaces.