Publishers move to block Google crawlers as AI search surges

For decades, Google has been the way the world navigates the internet. It has a global search engine market share north of 90%.

Even a couple of years ago, the idea that major publishers might consider blocking Google crawlers would have been hard to comprehend. On paper, it’s the ultimate own goal.

But the incredibly rapid rise of AI has fundamentally reshaped the search landscape in a short space of time. With Google yielding far more zero-click searches than ever before, and repurposing content for its own AI Overviews, the trade-off is no longer the no-brainer it once was.

USA Today is publicly contemplating the nuclear option: blocking Google’s access altogether. As other major publishers weigh up taking the same step, the very nature of how we browse online is at a hugely significant crossroads.

The Problem: Diminishing Google Search Returns, Unauthorized Content Repurposing

“It’s time to take a stand and say enough is enough.”

Those were the words of USA Today Chief Executive Mike Reed, via the Wall Street Journal. The company is contemplating turning off Google’s access to its content.

It would have been a significant statement from any major news brand. But when you consider that USA Today also owns major regional publications like Detroit Free Press and The Arizona Republic, along with UK subsidiary Newsquest, the implications are particularly far-reaching. And that’s before taking into account any potential domino effect.

The key to understanding how we’ve reached this stage is the rise of the zero-click search.

Searches for the term “zero-click” have increased by 900% in the last 5 years, with further growth forecasted.

A zero-click search is a search engine query that does not result in a user clicking through to a website. As Google has rolled out and improved upon its AI Overviews, the percentage of searches ending in no click has increased dramatically.

An Exploding Topics survey last year found that only 18.6% of people always or usually click through to the source of an AI Overview.

This increasing prominence of – and reliance on – AI Overviews creates two simultaneous problems for publishers: an economic issue of utility, and a bigger-picture issue of fairness.

Google’s diminishing value to publishers

Publishers rely on clicks to survive. Brands may be more or less insulated by virtue of the size of their paid subscription base, which can somewhat ease the dependence on advertising, but every media company ultimately needs eyes on their content.

The Google ecosystem was historically a win-win. Publishers opened their content up to the search engine’s crawlers, and were rewarded with vastly scaled-up discoverability.

The deal worked well when discoverability equaled clicks. But the “Great Decoupling” has fundamentally altered the equation, leaving publishers asking what they’re actually getting from Google in 2026.

UK publishing giant Reach, responsible for titles such as the Express and Mirror, is a case in point. Its latest results showed Google referral traffic was down by 55% YoY.

If Google isn’t delivering clicks, media outlets will begin to wonder exactly what it is delivering.

There is still major value in being cited as a source or mentioned by name in Google’s AI Overviews, and AI visibility is where many brands should now be focusing their efforts. For instance, E-commerce websites can derive huge benefits when AI recommends them to high-intent users.

“AI visibility” searches are up by 7,000% in the last 2 years.

But for publishers, their very product is the content that Google is synthesizing. The loss of a click directly correlates to a loss of business, which means that the bargain needs to be seriously reassessed.

Are publishers inadvertently contributing to their own demise?

Not only are publishers getting vastly reduced clicks via Google, they are actually freely improving the tool that is taking those clicks away. AI Overviews rely on access to third-party content to serve users with informed, up-to-date responses.

Where AI gets its information is a major cross-industry point of discussion. Anthropic recently reached a $1.5 billion class action lawsuit settlement, compensating authors whose books had been used to train Claude models.

For online publishers, the issue isn’t just training data. In order to give current answers in its AI Overviews, Google is continuously scraping content, including (with publisher consent) articles that have been put behind a paywall.

In other words, it is being given unlimited free access to what is typically a paid product (either in the form of a subscription, or via ad revenue). And it is using that access to disseminate the enclosed information to non-paying users.

You could liken it to someone subscribing to a paid newsletter, and then forwarding it on to hundreds of people. Except in this case, the “subscriber” (Google) isn’t even paying in the first place.

So is it fair that Google gets all of this content for its AI Overviews, free of charge? USA Today is starting to think that the search giant should have to pay, just like anybody else would:

“I wouldn’t call it a big decision because we’re blocking other crawlers,” Reed explained. “For those with licensing agreements, they get our content. For those without, we block them.”

Which Websites Block Google Crawlers in 2026?

It isn’t just USA Today considering a block on Google crawlers, or a re-evaluation of their relationship with the search giant.

Per the WSJ, Politico has discussed limiting Google’s access to its free articles, implementing a registration wall that blocks crawlers. Reuters has also considered blocking Google’s bots for its consumer-facing news articles, and People Inc. is considering a total block.

Meanwhile, Penske Media (owner of Variety and Rolling Stone) has a lawsuit pending against Google, arguing that AI Overviews are using its content illegally and reducing traffic as a result.

On the other hand, Financial Times executive Matt Rogerson has previously said that publishers don’t have a “genuine choice,” as long as opting out of Googlebot means losing search visibility as well as presence in AI Overviews.

An interesting dynamic has subsequently developed in the UK, where a regulatory authority has demanded that Google allows publishers to opt out of AI features, without sacrificing presence in traditional search. The Economist is among the brands looking into this option.

But which websites are already actively blocking Google crawlers?

To answer that question, it’s important to make the distinction between Googlebot and Google-Extended.

“Googlebot” searches are up by 271% in the last 5 years.

When publishers opt out of being crawled by Google-Extended, they can no longer be crawled by Gemini. But in most cases, when they opt out of Googlebot, they throw the baby out with the bathwater: they don’t appear in AI Overviews, but they don’t appear in Search either.

As a result, no major outlets are fully blocking Google yet:

Publisher	Blocking Google-Extended	Blocking Googlebot
BBC	Yes	No
Yahoo	Yes	No
New York Times	Yes	No
CNN	Yes	No
USA Today	Yes	Considering
Reuters	Yes	Considering
People	No	Considering
Politico	No	Considering

The list of publishers restricting the use of their content for AI training is far more extensive. Almost 80% of the biggest US and UK news websites are now blocking AI training bots.

The BBC, New York Times, New York Post, NBC News, Buzzfeed, Huffpost, and The Hill are all blanket-blocking AI training crawlers from the likes of Google, OpenAI, and Anthropic. They remain opted into Googlebot scraping.

And blocking AI crawlers is about to become much easier for the 24 million active websites that use Cloudflare. The cloud hosting platform made it easier to block AI crawlers in a single click last year, and that’s now changing to become the default option for agent and training bots.

Is it Really Time to Block Google?

For the vast majority of sites, it probably still isn’t the right call to block Google’s crawlers, and specifically Googlebot. Media outlets are in a unique position.

The average E-commerce website certainly still stands to make a net gain from Google, even in an age of AI Overviews. It’s all about AI readiness.

Just as it always has been, it’s clearly wise to put effort into building owned acquisition channels. Ones that are not reliant on Google, AI Overviews, ChatGPT, or anything else that can change at the drop of a hat or the tweaking of an algorithm by a third party.

But at the same time, it’s important to optimize for the current reality of how consumers are searching. That means making sure that your brand is discoverable by AI.

You can check your AI readiness using this free tool. It will show you whether AI crawlers can reach your site, and how much you’re being cited and mentioned in ChatGPT.

Even media sites will think extremely carefully before actually shutting the door on Google. Although their referrer-in-chief is sending substantially less traffic now, brands will have to be very confident in their other readership avenues to conclude that they can afford to turn off the tap.

Traffic driven from AI Overviews is substantially lower than the traffic that was previously driven from Google searches. But it’s better than no traffic at all (and Google says that the traffic it does drive is higher-quality, with readers staying on-page for longer.)

I’ve spent the best part of a decade in journalism, working both freelance and full-time for major media brands, and I have certainly encountered publishers who have effectively built their entire strategy around reliance on Google. Outlets in this position may decide they have little choice but to get themselves properly ready for the AI search era.

Are AI Overviews Here to Stay?

The even bigger question underpinning the whole blocking Google crawlers debate is whether AI search is here to stay.

There’s no sign that Google will be getting rid of AI Overviews any time soon. But there are indications of increasing consumer dissatisfaction, which might at least mean a plateau and eventual reversal of the zero-click searches trend.

Last year, 82% of consumers found AI Overviews more helpful than traditional search. That’s now fallen to 54%, per Fractl.

Here’s the analysis from Exploding Topics co-founder Brian Dean, who believes we’re entering an age of manual user fact-checking:

So change in consumer search behavior may yet prove to be a lifeboat for publishers. But although the landscape will keep evolving, brands cannot afford to blindly hope that everything will go back to the way it was before.

Optimize for AI, Don’t Rely on It

It will be fascinating to see if the publishing industry really does move en masse to block Google’s crawlers. That could prompt a seismic change in how we navigate the web.

But for the vast majority of sites, the formula is clear: optimize for AI, while making sure you have other acquisition channels too.

If you hitch your wagon to AI entirely, you might one day find yourself in the exact same boat as the publishers who thought Google traffic would never go away.

If you pretend that AI hasn’t already fundamentally changed how users discover websites and brands, you’re leaving audience on the table, and risk being left behind by more agile competitors.

The AI Agent Readiness Checker tool is a great place to start. And if you want to build a full AI strategy, something like Semrush Enterprise will help you to place your brand front and center of AI search.

Whatever you do, the age of AI search is impossible to ignore. For publishers, E-commerce brands, and everything in between, we are entering a brave new world, one where Google search is no longer the only game in town.