In one sentence
An AI crawler is a bot, like GPTBot, ClaudeBot or PerplexityBot, that reads your site so an AI company can train its models or answer shoppers' questions, and blocking it can hide your store from AI.
How AI crawlers read your store
An AI crawler requests your pages the way a browser does, reads the text it gets back, and takes it to the AI company's systems. Each one identifies itself with a name, called a user-agent, so your robots.txt file and your firewall can tell them apart.
Many AI crawlers read the HTML your server sends and don't run JavaScript. Product details that only appear after scripts load, like size charts in a tab or reviews from an app, may never reach them. See How to check if AI can read your product pages.
AI training crawlers vs. AI search crawlers
Most AI companies run separate bots for separate jobs. Which one you block changes what happens.
- OpenAI · Training crawler: GPTBot · Search crawler: OAI-SearchBot · Fetches a page when a user asks: ChatGPT-User
- Anthropic · Training crawler: ClaudeBot · Search crawler: Claude-SearchBot · Fetches a page when a user asks: Claude-User
- Perplexity · Training crawler: None listed · Search crawler: PerplexityBot · Fetches a page when a user asks: Perplexity-User
- Google · Training crawler: Google-Extended (a robots.txt setting, not a separate bot) · Search crawler: Googlebot · Fetches a page when a user asks: n/a
Training crawlers shape what a model knows in future versions. Search and user crawlers decide whether an assistant can read your pages when a shopper asks today.
Why AI crawlers matter for AI visibility
If an assistant's search crawler can't reach your pages, it can't name you from them. It falls back on what others say about you, or on nothing.
Blocks are often accidental: an old robots.txt template, a security app, or a CDN bot setting switched on for everyone.
How to check which AI crawlers can reach your store
- Open
yourstore.com/robots.txtand look for the names in the table above. - Check your CDN, firewall or bot-protection settings for AI bot blocking. See How to check if your security settings block AI crawlers.
- Decide bot by bot: training is a business choice, search decides whether you get recommended.
- Check again after theme, app or hosting changes.
AI crawler FAQs
Should I block AI crawlers on my store?
Block training crawlers if you don't want your content in future models. Leave search and user crawlers open if you want assistants to recommend you. Blocking everything keeps you out of both.
Do AI crawlers slow down my site?
They add load like any bot. If one hits too hard, rate-limit it at your CDN instead of blocking it outright, so you stay readable to assistants.
Can I see which AI crawlers visit my store?
Yes, in your server or CDN logs, filtered by the user-agent names above. Hosted platforms like Shopify don't show raw logs, so use your CDN's bot analytics if you have one.
About the author
Jenn Starr, Co-founder, Selix
Jenn has spent 20+ years helping SaaS startups build their marketing and implementation playbooks. She has bootstrapped three marketing agencies and is a 500 Startups alum. At Selix, she leads the work of getting ecommerce brands named in AI answers.
Related terms