In one sentence
robots.txt is the file at the root of your site that tells crawlers, including AI crawlers from ChatGPT and Perplexity, which pages they may read, so it decides whether AI can see your store.
How robots.txt works
The file lives at yourstore.com/robots.txt. Each rule names a crawler, called a user-agent, and the paths it may or may not read. Well-behaved crawlers check the file before they crawl.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
This example keeps OpenAI's training crawler out and lets its search crawler in. robots.txt is a request, not a lock. It doesn't hide pages from people, and not every bot follows it.
AI crawlers in robots.txt: training bots vs. search bots
Training bots shape what a model knows. Search bots decide whether an assistant can read your pages live when a shopper asks. See AI crawler.
Why robots.txt matters for AI visibility
One line can make your store invisible to an AI assistant. Blocks often come from a template copied years ago, an app, or a security setting nobody remembers turning on.
If an assistant's search crawler can't read your pages, it can't name you from them. That caps your AI visibility before any content work begins.
How to check your robots.txt for AI crawlers
- Open
yourstore.com/robots.txtin a browser. - Look for AI crawler names, and for
User-agent: *paired withDisallow: /. - Decide crawler by crawler: training bots are a business choice, search bots decide whether you get recommended.
- Check your CDN or firewall bot settings too. Some security tools block AI crawlers even when robots.txt allows them.
- Check again after theme, app or hosting changes.
robots.txt FAQs
Should I block GPTBot on my store?
It's a business choice. Blocking GPTBot keeps your content out of future OpenAI training, but it doesn't remove you from ChatGPT search, which uses OAI-SearchBot. If you want to be recommended, leave the search crawlers open.
Can I edit robots.txt on Shopify?
Yes. Shopify creates a default file, and you can customize it by adding a robots.txt.liquid template to your theme. Edit carefully: a wrong rule can block Google too.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews are part of Google Search, which uses Googlebot. Google-Extended only controls whether your content is used for Gemini models.
Is robots.txt the same as llms.txt?
No. robots.txt tells crawlers what they may read. llms.txt is a proposed file that points AI tools to your most useful pages. robots.txt is the one crawlers check today.
About the author
Jenn Starr, Co-founder, Selix
Jenn has spent 20+ years helping SaaS startups build their marketing and implementation playbooks. She has bootstrapped three marketing agencies and is a 500 Startups alum. At Selix, she leads the work of getting ecommerce brands named in AI answers.
Related terms