Vercel Firewall Now Lets You Block AI Crawlers and AI Assistants Separately
Vercel has updated the AI Bots managed ruleset in Vercel Firewall so that requests from AI crawlers and AI assistants can be logged or blocked separately. Site owners can, for example, keep training and indexing bots out while still letting assistants fetch their pages to answer user questions.
- Vercel Firewall's AI Bots managed ruleset can now log or block AI crawlers and AI assistants independently.
- AI crawlers collect content for indexing and model training; AI assistants fetch pages in response to user actions or search queries.
- Settings are available in the dashboard under Security > Firewall > Rules, or via the Vercel CLI.
- Vercel says it updates the rules as new bots are added.
- The AI Bots managed ruleset is available on all plans.
Two bot categories, two separate actions
According to Vercel, the AI Bots ruleset now distinguishes between two types of automated traffic. AI crawlers are bots that collect content for indexing and model training. AI assistants are bots that fetch pages in response to user actions or search queries.
Each category gets its own action, either log or block. Vercel gives one example of what this allows: a project can block AI crawlers while allowing AI assistants to retrieve its content to answer users' questions. Vercel also says it updates the rules as new bots are added, so teams do not need to maintain bot lists themselves.
How to configure it
In the dashboard, the controls sit in a project's Security > Firewall > Rules section. Under Bot Management, users enable AI Bots and select the gear icon beside it. They then set an action for AI crawlers and for AI assistants, review the changes and publish them.
The same settings can be changed with the Vercel CLI. Vercel's examples use these commands:
- vercel firewall rules edit ai-bots --category ai-crawlers --action deny --yes to block AI crawlers
- vercel firewall rules edit ai-bots --category ai-assistants --action log --yes to log AI assistant requests
After editing, users can review the pending changes with vercel firewall diff and apply them with vercel firewall publish.
What it means for site owners
Until now, the distinction Vercel describes had to be handled by the site owner. The split gives teams a way to treat bulk collection for training and indexing differently from on-demand retrieval triggered by a person asking a question. A site could, for instance, decline to be used for model training while remaining reachable when an assistant looks up its pages for a user.
The AI Bots managed ruleset is available on all plans, according to Vercel, so the controls are not limited to paid tiers. The announcement lists Sage Abraham as a contributor, and further details are in Vercel's documentation.
