A plain-English series AI & Your Business
Part 4

AI & Your Business, Part 4. A plain-English series for business owners who already use AI in their day. Part 1 was about AI that keeps working while you sleep, Part 2 about running your business from one conversation, and Part 3 about Shopify stores showing up in AI shopping answers. This one is for anyone with a website: who’s knocking, and who gets in.

When a customer asks ChatGPT, Perplexity, Claude or Google for a recommendation, a bot may go and read websites to answer. Sometimes it’s a search bot building an index. Sometimes it’s an agent opening your page because a person asked it to, right then.

Your website decides whether those bots get in, sometimes without anyone at your business choosing. A file called robots.txt, a security setting at your host, a pop-up, a login wall: each one can quietly let the good bots in or keep them out. So “AI bot ready” mostly means making that choice on purpose.

The short versiontap to open or close
  • AI companies name their bots by job. OpenAI, Google, Anthropic and Perplexity each publish separate names for search, for training and for fetching a page when a person asks.
  • Block every bot and you can drop out of the answers. Turning away the search bots can keep you out of ChatGPT, Perplexity and Claude search results.
  • Training is a separate choice. Saying no to the training bots doesn’t remove you from search, according to OpenAI and Google.
  • Say it in text. Prices, stock, hours, policies and contact details belong on the page as words, with real buttons and structured data, not hidden behind logins or captchas.
  • Keep the bad bots out at the doors that matter: checkout, forms and logins. A bot’s name can be faked, so the good ones publish their addresses.

Who’s knocking: one name per job

OpenAI, Perplexity, Anthropic and Google each publish the names of their bots, and each name does one job:

The jobOpenAIPerplexityAnthropicGoogle
Search: builds the index AI answers draw fromOAI-SearchBotPerplexityBotClaude-SearchBotGooglebot
Training: whether your pages may train AI modelsGPTBot(none named)ClaudeBotGoogle-Extended
A person’s request: opens a page because someone askedChatGPT-UserPerplexity-UserClaude-UserGoogle-Agent

A few details from each company’s own pages:

  1. OpenAI says OAI-SearchBot is used to surface websites in ChatGPT’s search features, and that sites opted out of it “will not be shown in ChatGPT search answers.” Its settings are independent: you can allow OAI-SearchBot and disallow GPTBot. Changes can take about 24 hours.
  2. Perplexity says PerplexityBot surfaces and links websites in its search results and isn’t used to train AI models. It recommends allowing PerplexityBot in robots.txt.
  3. Anthropic says its bots honor robots.txt. Turning away Claude-SearchBot may reduce your visibility in Claude’s search results; restricting ClaudeBot signals your future content should stay out of training.
  4. Google says Google-Extended controls whether Google may use your content to train and ground its Gemini models, and that it “does not impact a site’s inclusion in Google Search.” To appear in AI Overviews and AI Mode, Google says a page only has to be indexed and eligible for a snippet in Search, with “no additional technical requirements.”

The third row is the newest. When a person asks an assistant to check your hours or compare two products, it sends a fetcher on their behalf. OpenAI says robots.txt rules “may not apply” to ChatGPT-User, Perplexity says Perplexity-User “generally ignores robots.txt rules,” and Google says the same of the user-triggered fetchers that include Google-Agent, which Google uses for agents that navigate the web and take actions for a person. Anthropic’s Claude-User is the exception: it follows robots.txt like Anthropic’s other bots.

Let search in, then decide on training

Your robots.txt file lives at the root of your site (yoursite.com/robots.txt). It tells bots which pages they may read, one group per bot name. A bot that isn’t named follows the group for every bot, written User-agent: *.

Here’s one example, for a business that wants to show up in AI search but would rather its pages not train AI models. It’s a choice, not a recommendation:

User-agent: *
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Everything else, Googlebot and the AI search bots included, falls under the first group and is allowed. Two cautions from Google’s own robots.txt guide: the file is “not a mechanism for keeping a web page out of Google” (that’s what noindex or a password is for), and its rules “cannot enforce crawler behavior.” Well-behaved bots follow it. Bad ones don’t.

Check the lock you didn’t know you had

robots.txt is the sign on the door. Your host or security plugin is the lock, and it can turn a bot away before it ever reads the sign. OpenAI recommends allowing requests from its published IP ranges, and Perplexity says a site behind a firewall may need to explicitly allow its bots, checking both the bot’s name and its address.

So if you use a firewall, a security plugin or a bot-blocking setting at your host, make sure it isn’t turning away the search bots your robots.txt invites.

Say it in text, not in pictures

Getting in is half of it. The other half is what a bot can read once it’s there. Google’s advice for its AI features is to make sure “important content is available in textual form” and that your structured data matches what’s visible on the page. Google’s guide for agent-friendly sites adds that AI agents read a page through screenshots, the raw HTML and the accessibility tree.

In plain terms:

  1. Put the facts in words. Prices, what’s in stock, hours, address, phone, service areas, shipping and returns, written on the page, not only inside a photo or a flyer image.
  2. Label them with structured data. Product markup carries price, availability, shipping and return details; LocalBusiness markup carries your hours, address and phone. (Google stopped showing FAQ rich results on May 7, 2026, so write your answers on the page; the words do the work.)
  3. Use real buttons and links. Google’s agent guide recommends real button and link elements over look-alikes, and warns against transparent overlays that hide what an agent needs to click.
  4. Don’t wall off what you sell. Information behind a login is out of reach: Google lists password protection as a way to keep a page out of its results. Anthropic says its bots “will not attempt to bypass CAPTCHAs.” Keep logins and captchas on accounts, forms and checkout, not on the pages that describe your products and services.

If this sounds familiar, it’s because these are the same basics as five things that get you found on ChatGPT and Gemini. Bots and people want the same thing: a clear page.

What about llms.txt?

llms.txt is a short summary file for AI tools, proposed in September 2024 and still a proposal. Google says Google Search ignores these files, and that having one “will neither harm nor help” your rankings there. It’s still fine to keep one for other tools that read it. Shopify stores now include one by default.

Our take: nice to have, cheap to add, and never a substitute for the steps above.

Good bots, bad bots

Not every visitor calling itself Googlebot is Google. A bot’s name is just a line of text, and Google’s own guide to verifying its crawlers is written for owners “concerned that spammers or other troublemakers” are claiming to be Google. That’s why the good ones publish where they come from:

  1. Google explains how to confirm a visitor through a reverse and forward DNS lookup, and publishes its IP ranges.
  2. OpenAI and Perplexity publish the IP addresses for each bot.
  3. Anthropic publishes the addresses its bots come from, and says blocking by address may not work as a reliable opt-out. Use robots.txt for that.

The practical rule: guard the doors where bad bots do damage (checkout, contact forms, logins, account sign-ups), and leave the pages that describe your business open to the bots you want.

For online stores

If you sell on Shopify, a lot of this is already handled:

  1. Bot management is built in. Shopify says every request to your store passes through Cloudflare first, and that bot management “is handled directly for stores on Shopify, and you don’t need to take any action.” hCaptcha is on by default for contact forms, comments and customer accounts.
  2. robots.txt editing is for experts. Shopify’s default file “works for most stores,” and editing it is “an unsupported customization” that can cost you “all traffic” if done wrong.
  3. AI shopping doesn’t depend on robots.txt. If you sell through Shopify’s agentic storefronts (the subject of Part 3), Shopify sends your product data to those channels “independently of /robots.txt.” Blocking AI crawlers only affects how bots find you on the open web.
  4. Checkout protection is a Plus feature. Shopify’s checkout bot protection, which blocks known bots during busy sales, is available on Shopify Plus.

The owner’s checklist

  1. Open yoursite.com/robots.txt and read it. Make sure it doesn’t block Googlebot, OAI-SearchBot, PerplexityBot or Claude-SearchBot.
  2. Decide on training separately: GPTBot, ClaudeBot and Google-Extended. Either answer is fine; just make it on purpose.
  3. Check your firewall, security plugin or host settings for a bot blocker that overrules robots.txt.
  4. Put prices, stock, hours, policies and contact details on the page as text.
  5. Add Product and LocalBusiness structured data that matches what’s visible.
  6. Use real buttons and links, and keep pop-ups from covering them.
  7. Keep logins and captchas on checkout, forms and accounts, not on product and service pages.
  8. Add an llms.txt if it’s easy, and don’t expect it to do the work.

How we help

This is part of our AI search work (Answer Engine Optimization). We check what keeps AI tools from reading and naming your business, then fix it: the bots your site lets in, the facts on the page, and the structured data that labels them. For Shopify stores, we start with the AI shopping readiness check from Part 3.

AI is already knocking. Let’s make sure the right ones get in.

Is Your Website Ready for AI?

Prefer to talk it through first? Let’s talk.

Sources

All read October 20, 2026.

  1. OpenAI, Overview of OpenAI Crawlers
  2. Perplexity, Perplexity Crawlers
  3. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?
  4. Google Search Central, AI features and your website
  5. Google Search Central, Google’s common crawlers (Google-Extended)
  6. Google, Google user-triggered fetchers (Google-Agent)
  7. Google Search Central, Introduction to robots.txt and Create a robots.txt file
  8. Google Search Central, Verifying Googlebot and other Google crawlers
  9. Google Search Central, Product and LocalBusiness structured data
  10. Google Search Central, Optimizing your website for generative AI features (llms.txt and AI agents) and documentation updates (FAQ rich results retired)
  11. web.dev (Google), Build agent-friendly websites
  12. llmstxt.org, the llms.txt proposal
  13. Shopify Help Center, Editing robots.txt.liquid (controlling AI crawler access), Dealing with bots and Bot protection
  14. Shopify developer changelog, Customize /llms.txt, /llms-full.txt and /agents.md

How we sourced this: every statement about a bot, a setting or a platform comes from the company’s own page listed above, read October 20, 2026. Bot names and rules change, so check those pages before you edit your own robots.txt. The robots.txt sample is one example of a choice, not a recommendation for every business.