Most of the audience on the Internet today are machines.
Ask the chatbot a question and observe what is happening with the web space hiding behind it. To form an answer, the bot “reads” 30-40 pages, selects the necessary information and gives you a concise, neat paragraph. You don’t see these pages or follow the links to them. The site that “won” (whatever that word means today) gets only a mention in pale gray font, and not a single real visitor.
This is how most of the content consumption on the Internet occurs today, and companies managing network infrastructure see this picture firsthand. This year, on the Cloudflare network, the number of requests to web pages from bots exceeded the number of requests from humans: 57.5% versus 42.5%. The company’s CEO noted this turning point back in June, about a year and a half earlier than his own forecast, linking such a surge in activity to the work of software agents loading pages on behalf of users.
The artificial intelligence segment is growing faster than all others: last year, its growth rate was about eight times higher than the growth rate of human traffic. Web content is being read more actively than ever. But it’s not people who read it.
The rules that you and I used to build web rules regarding quality, accessibility, and honesty were originally written for people. But the audience has changed. And the rules are now being quietly abolished, without any votes, adapting to the new realities. Three of them are already a thing of the past…
- Who keeps order on the web and why do they need it
- Who pays for access
- What is considered fraud today
- What’s left for people
Who keeps order on the web and why they need it
For more than twenty years, the web had its own “regulator”, and that regulator was Google. As soon as you stuffed the text with invisible keywords, bought a thousand links, or created doorways, sooner or later you would be punished from above. To most, it seemed that these were just “sanitary measures”: Google, they say, out of pure altruism, brings order to the network for the benefit of all users.
In fact, everything was different. Google kept order on the Web because it sold ads linked to the search index, and an index filled with garbage is valued lower by advertisers. This “cleaning” was akin to putting polish on a shop window.
Andy Baio observed a similar pattern ten years ago: Google allowed Google Books and its news archive to fall into disrepair as soon as they stopped generating revenue, and warned against attempts to entrust the corporation with library functions. However, he was also condescending to them. The library has always been just a by-product of the advertising business: it was supported as long as it brought in money, and abandoned when it stopped.
And now take a look at the new audience. The system that provides ready-made answers (answer engine) does not sell ads based on an ordered index, because it does not store it in a form that can be viewed by the user. She reads the information, evaluates what she finds, and reproduces only those fragments that she considers necessary. A weak page is not penalized — it is simply not selected for display. From the point of view of the page itself, this is even worse: after all, when a fine was imposed, at least an email notification was received. There is no “supervisor” anymore. All that’s left is the doorman, who never explains why you weren’t allowed inside.
And the funny thing here is that Google now combines both of these roles. The company still manages an index that is monetized through advertising, the same one that the court recently recognized as an illegal monopoly (both in the field of search and in the sale of related advertising). At the same time, she creates a response system that makes this index unnecessary, and immediately crams advertising into it as actively as the format allows: sponsored images mixed with image search results; advertising inside AI-generated summaries; and a full-fledged online trading channel attached to the side. The company protects its old business by creating a product that will ruin it, and sells ads on this “murder weapon.”
Who pays for access
The previous scheme was a fair deal—and even very generous. “Launch our search robot for free,” Google said, “and we’ll send readers to you.” Website owners didn’t just put up with scanning — they fought for their sites to be scanned faster and indexed deeper, because this was the way to reach an audience.
Scanning for the needs of AI does not offer anything like this. The system takes the same content, processes it into a ready-made response, and returns absolutely nothing in return: no click, no reader, no opportunity to sell ads.
In July 2025, Cloudflare, through whose infrastructure about a fifth of all web traffic passes, openly announced what they previously preferred to keep silent about: the previous model is no longerworks. Now, new sites on the Cloudflare network block AI robots by default, and owners can charge a fee for each scan through a special marketplace; any bot who does not want to pay receives a polite “payment required” notification — and nothing more. Thirty years of persuading Google to scan websites more actively — and now the natural reaction is to install a toll gate.
In the UK, 31 content publishers went beyond simple blocking. They turned the old file into robots.txt — this polite “please don’t”, which search robots have learned to ignore, is included in a legally binding contract: by downloading the page and using the article without payment, you thereby agree to issue an invoice of 500 pounds, which can be collected through the court in the same way as any other debt.. So far, no one has been able to actually recover this money from OpenAI, but such a step clearly shows where everything is going. Access to the content has a price tag.
Why now, and not five years ago, when mass data collection (scraping) began? Because texts have become a scarce resource. When the quality of a model directly depends on the texts it learns from, and the open Internet is filled with the “exhaust” of the model itself, authentic author’s texts cease to be just raw materials and turn into a valuable asset. The content owners realized that they are the ones who control the incoming data, and the data has a price.
So keep an eye on what they are actually doing, because it is in the actions that the essence of the dispute lies. The New York Times is suing OpenAI for using its archive to train neural networks and at the same time licensing the same archive to Amazon.
OpenAI has already entered into agreements with The Guardian, The Atlantic, The Washington Post, News Corp and a number of other publications. No one on this list is trying to drive cars away. They only haggle about the price. The genie has already been released from the bottle, the demand is structural, and there has never been a debate about whether this will happen at all. The only question is who gets what for it. The road will be built anyway; the only issue is the amount of the toll.
What is considered fraud today
There is a rule that is older than all others: never show a search robot content other than what the user sees. This is called “cloaking”, and for cloaking the site is removed from the search results. Every SEO specialist finds out about this on the very first day of work.
But it’s worth referring to the official definition of Google, not to popular myths. Cloaking is the demonstration of various content to users and search engines in order to manipulate positions in search results and mislead people. It was not the fact of the difference in versions that made this a violation, but a hidden intention to deceive. To show a search robot a vacation page, and to show a human a page where fake medicines are sold, is cloaking. But providing the same information in a cleaner, machine—readable form, to a system that accepts only such a format, is not cloaking and has never been considered it. That’s exactly what Google’s recommendations say: change the way you serve as much as you want, as long as the essence remains the same.
Therefore, if the “reader” is an AI agent who needs structured data, and not your title image or a cookie notification banner, then providing him with structured data is not a trick. This is a response in the language in which the request was made. The taboo arose from the assumption that a search robot is just a substitute for the human eye. Once this assumption is dropped, most of the taboos lose their meaning.
Here I have to be a nerd for a moment, remembering the days of fighting web spam, because the line still exists — it just shifted. If you lie to a machine in order to improve your position or feed it something that you would never subscribe to in a conversation with a living person, this will still be regarded as an intention to deceive. And in areas related to health, finance, or any other areas where an incorrect answer can harm someone, it is also dangerous. The border runs exactly where the deception begins. Formatting content for a reader who happens to be a machine has never been a violation of this boundary.
What people have left
None of this is a forecast for the future. This has already happened to those segments of the Internet that are most important to machines, and the process is spreading further. The direction of development is beyond doubt. The only open questions are about the terms: who will receive the money, who will be cut off from access, and whose business model will win the right to determine what a high-quality response should be.
The last question remains completely open, and it should be closely monitored, since the creators of these new “entrance gates” to the Internet cannot yet agree on how to make money from them.
Google embeds ads directly into the responses. Perplexity experimented with advertising, but rejected it, insisting that the user should believe that he gets the best response, not the one for which he paid more. Anthropic keeps Claude ad-free and loudly proclaims it. OpenAI is testing ads, promising that they won’t affect the content of responses, which is exactly what Google once promised about search results, and we all remember how it turned out.
Whoever wins this dispute will inherit the role of “supervisor” and gain the right to define the concept of “good” for the Internet, which most people now perceive only indirectly.
For all of us, this leaves a different, more unusual task. The machine-reading internet doesn’t care about witty headlines or eye-catching page designs. He cannot be charmed or cajoled by flattery.
The machine selects only what is useful for forming a response, and discards everything else. This means that only the essence hidden behind the external design retains value: original materials and the ability to evaluate the correctness of an answer even before it gets to a person who is ready to act on it.
For thirty years, the Internet has been learning to work for people. Now he has to become useful to someone who has neither thumbs to show “class”, nor palms to applaud.