Cloudflare has launched a new feature named the pay-per-crawl system to enable content creators to have effective means of controlling access to sites by AI bots. So far, the only choices available to the publishers were to enable free takeover of its content by all AI crawlers or to block it completely. Under this new model, the third option is to get paid when AI crawlers access the content.
This system is constructed in the HTTP 402 code referred to as Payment Required that was implemented many years ago and is seldom seen. In the situation when an AI bot attempts to access a page, it should provide a payment proposal with the request. Access is provided when the offer corresponds with the specified price of the content owner. Otherwise, they receive 402 responses with the price accompanied by the server. The bot will then assent in paying or wave off.
The payment system is undertaken by Cloudflare, which serves as the middleman. The site owners have the option to decide on what to do with every bot: blocking them, enabling them, or charging them. Although a bot may not have a payment arrangement with Cloudflare yet, there is a mechanism by which the content owner can declare his content as chargeable, effectively presenting a block yet leaving a loophole to future agreements.
It is all about security. Bots need to register, show their public keys and sign every request, thus preventing imitation. The paid requests can only be made by the bots authenticated successfully. This will keep out spoofed crawlers and will make sure we are charging the correct bot.
This system is currently in private beta, although Cloudflare is optimistic that in future, it can be used to license content. In the long term, the publishers might even have the ability to charge variably priced content or provide complete paywalls to agents and bots.
This movement is not all about money. It is actually to reclaim control of the content creators. Rather than companies interested in AI snatching data without consulting anyone, this creates a model in which they pay for the value they receive. With the internet dawning an Era of AI, several tools can aid creators, such as pay-per-crawl, to avoid trouble.
Many publishers, content creators and website owners currently feel like they have a binary choice either leave the front door wide open for AI to consume everything they create, or create their own walled garden. But what if there was another way?