Which Brands Block AI Crawlers in robots.txt? We Checked 3,589 Domains

Every major AI company sends crawlers to index the web. OpenAI uses GPTBot. Anthropic uses ClaudeBot. Google uses Google-Extended. Perplexity uses PerplexityBot. ByteDance uses Bytespider.

But do brands actually block them? We crawled the robots.txt files of 3,589 domains in the Ataiva AI Brand Index to find out. This is proprietary data from Ataiva's crawl infrastructure — nobody else has this dataset.

The Big Picture

Of 3,589 brands we crawled, 40+ brands actively block at least one AI crawler. That's roughly 1% of domains taking an explicit stance. The other 99% either allow all AI crawlers or don't address them in robots.txt at all.

But the interesting story isn't the overall number — it's which bots get blocked, and by whom.

Notable Brands Blocking AI Crawlers

Brand	Domain	GPTBot	Google-Ext	ClaudeBot	PerplexityBot	Bytespider
Amplitude	amplitude.com	❌	❌	❌	❌	✅
Canva	canva.com	❌	✅	❌	❌	❌
Amazon	amazon.com	❌	❌	❌	❌	❌
eBay	ebay.com	❌	✅	❌	❌	❌
Figma	figma.com	❌	❌	✅	✅	✅
Monday.com	monday.com	✅	❌	❌	❌	✅
Squarespace	squarespace.com	❌	❌	❌	✅	❌
Bitrix24	bitrix24.com	❌	❌	❌	✅	✅
SAP	sap.com	❌	❌	❌	❌	✅
Retool	retool.com	❌	✅	❌	❌	✅
Indeed	indeed.com	❌	❌	✅	✅	❌
Glassdoor	glassdoor.com	❌	❌	✅	✅	❌
Fortinet	fortinet.com	❌	❌	✅	❌	✅
Bitwarden	bitwarden.com	❌	✅	❌	❌	❌
FreshBooks	freshbooks.com	✅	✅	❌	✅	❌

✅ = Allowed   ❌ = Blocked. Data from Ataiva's April 2026 crawl.

Key Findings

Amazon blocks everything

Amazon blocks all five AI crawlers. GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and Bytespider are all disallowed. Despite this, Amazon still scores 94 in AI visibility — because AI models were trained on years of data before these blocks existed. Blocking crawlers today doesn't erase your presence from models already trained.

Amplitude runs the tightest ship

Amplitude blocks GPTBot, Google-Extended, ClaudeBot, and PerplexityBot — only allowing Bytespider. This is one of the most aggressive blocking stances we found among SaaS companies.

Figma takes a selective approach

Figma blocks GPTBot and Google-Extended but allows ClaudeBot, PerplexityBot, and Bytespider. This suggests a deliberate strategy rather than a blanket block — possibly related to specific licensing or partnership considerations.

Monday.com blocks selectively too

Monday.com allows GPTBot and Bytespider but blocks Google-Extended, ClaudeBot, and PerplexityBot. Interesting given they score 99 in AI visibility — they may be strategically allowing OpenAI's crawler while blocking others.

Cybersecurity brands are split

Fortinet blocks GPTBot and PerplexityBot but allows ClaudeBot. Bitwarden blocks GPTBot but allows Google-Extended. There's no industry consensus — each brand is making independent decisions.

Does Blocking AI Crawlers Hurt Visibility?

Not yet — and that's the key insight. Amazon blocks all AI crawlers but still scores 94. Bitwarden blocks GPTBot but scores 94. Current AI models were trained on data collected before most of these blocks were implemented.

But this will change. As models are retrained on fresh data, brands that block crawlers may see their AI visibility decline over time. It's a slow-moving risk that most brands aren't tracking yet.

Check Your Brand's Crawler Status

Want to know if your competitors are blocking AI crawlers — or if you should be? Every brand in the AI Brand Index includes crawler access data alongside visibility scores.

Start tracking your AI visibility to monitor how crawler blocking decisions affect your brand's presence in AI recommendations. Or use our free tools to check your brand right now.