Cloudflare launches a web search API for AI agents in partnership with Exa and Ceramic
On October 2, Cloudflare introduced Web Search API, a service that lets AI agents search the internet for current information instead of guessing the address of a page. It comes through AI Gateway, the Cloudflare service that brings together control, logs and billing for calls to AI models in one place, and it starts with three search providers: Ceramic.ai, Exa and Linkup.
It is in open beta: it can be tried today, but it is not the final version yet. Searches are paid with the AI Gateway credit balance, at each provider's list price and with no markup, or with your own key for each provider. It works through a REST API or from Workers, the Cloudflare platform for running code, and Cloudflare says it will soon add it as a built-in tool in AI Gateway.
What interests us most is the condition set for the partners: their crawlers must meet Cloudflare's rules for verified bots, and every result must include a link to the page the content comes from. According to Cloudflare, these providers identify their crawlers, respect the robots.txt file (where each website states what can be crawled) and give the source of the results.
We think this is an interesting step for anyone with a website, because the search providers these agents use commit to respecting its rules and linking to its page. And for those who build agents, it is a way to give them up-to-date information without depending on them guessing an address correctly.
