GTM Engine · · 6 min read
AI bot traffic can skew your GTM numbers. Here is how to read them
By Harshil Jani
AI bot traffic is software from AI companies, not people, asking your site for pages, to train AI, power AI search or answer a chatbot user. It is smaller than you might expect, yet it can skew your go-to-market numbers, and the obvious fix, blocking it all, has a cost of its own.
In this article
- What one developer's server showed after a Hacker News post.
- How much web traffic is bots, and how little is AI.
- How to count buyers, and which AI bots to let in.
Where does AI bot traffic hit your GTM strategy?
It breaks the last step, measuring what worked, and the error flows back up. In my go-to-market order you pick the buyer, positioning, price, one channel and sales motion, then measure (what go-to-market actually is). If a third of your visits are bots, a launch looks bigger and conversion looks smaller, so you may back the wrong channel.
An analogy, not data: a shop that counts delivery drivers as shoppers sees great footfall, and a worse share of shoppers who bought.
The obvious suspect is AI. One developer's site shows why that is only half right.
What happened to the site that hit Hacker News?
One non-human visitor made nearly half of a small site's traffic. Sammy Straus's write-up "I posted to HN, and then my website was flooded with AI bot traffic" went up on Hacker News, a tech forum, on 13 October 2026. It passed 200 points (upvotes) within hours (Hacker News). After an earlier post about his feed reader, the site got "really, painfully slow" with only "a good handful" of active users (Sammy Straus).
It turns out that nearly half of our traffic was coming from one specific user.
Sammy Straus Developer of the feed reader sprinklz.io
That visitor calls itself a service that summarizes website changes into an AI newsletter, and it hit his site "many many times per-minute". A majority of requests did not run his JavaScript, the code that draws the page, so they were very unlikely to be people. He also saw named bots ("Claude bot, OpenAI bot", Google's search bot) and scanners probing for security holes.
He blocked the worst offender and added rate limits (a cap on requests per visitor). He also wrote a robots.txt, a text file telling bots which pages they may visit, to discourage AI bots but allow search.
An analogy for robots.txt: it is a "please knock" sign. Polite visitors obey; a bot can ignore it.
Was his site unusual, or is the wider web like this?
How much of web traffic is bots now?
More than half, by Cloudflare's count, and the twist is how little of that is AI. More than 20% of the web sits behind Cloudflare's network, it says. In September 2026 it wrote that "this year, for the first time, more than half of Internet traffic wasn't human" (Cloudflare, 2026). Cloudflare sells tools to block and charge AI bots, so it has a stake in how big this looks, and it sees only its own network.
You might expect most of that to be AI. Cloudflare's Radar dashboard, which counts web page requests, says otherwise.
Who requested web pages on Cloudflare's network, four weeks to 14 October 2026
AI bots were a single-digit share; non-AI bots made most page requests. These are requests, not visitors, and one bot can make thousands. Cloudflare also says daily AI agent requests grew more than 1,700% in a year.
The sharper question is whether AI bots send anyone back.
Which AI bots send buyers back?
Search engines send far more people back for each page they read. Cloudflare counts the pages a company's bots fetch for each visitor who clicks through to you from its product.
| Company | Pages fetched per visitor sent back, last 4 weeks | Bots in that count |
|---|---|---|
| 4.8 to 1 | Googlebot, GoogleOther | |
| Microsoft | 41.6 to 1 | Bingbot |
| OpenAI | 472.7 to 1 | GPTBot, OAI-SearchBot, ChatGPT-User |
| Anthropic | 501.9 to 1 | ClaudeBot, Claude-SearchBot, Claude-User |
| Perplexity | about 3,300 to 1 | PerplexityBot, Perplexity-User |
Source: Cloudflare Radar, Cloudflare's network only, read 14 October 2026. These numbers jump around (the seven-day view that day put OpenAI near 1,700 to 1), so read them as rough scale.
Traffic was always a poor proxy for value.
Matthew Prince Co-founder and CEO, Cloudflare
Prince sells the fix too, but the GTM point stands: on these numbers, few pages fetched by AI companies' bots send a buyer back. They can land in your reports, though.
How do bots mess up your numbers?
It depends on where you count. Google Analytics 4 (GA4), Google's visit-counting tool, removes "known bots and spiders" automatically. But you "cannot disable known bot traffic exclusion or see how much known bot traffic was excluded" (Google Analytics Help).
GA4 counts visits through a script in the browser, so a bot that skips that code, like most requests in Straus's logs, does not reach it.
Extra traffic lands in server logs (your server's record of requests), hosting dashboards and CDN analytics (a CDN delivers your pages, as Cloudflare does). Some of the gap is people, since visitors who block trackers or decline cookies can go missing from GA4.
A made-up example: 1,000 visits and 20 signups is 2% conversion. If 600 visits were bots and all 20 signups were people, the human rate was 5%. Judge on the first number and you might drop a channel that works.
Cleaning the count is half the job. Choosing who gets in is the other half.
Which AI bots should you let in, and which should you block?
Decide by the bot's job, using the names each company publishes. A request normally carries a user agent, a name tag for the program that sent it, which robots.txt rules use. A bot can fake it, so treat it as a claim.
| Bot name | Company | What it does | If you block it | My call |
|---|---|---|---|---|
| GPTBot | OpenAI | Collects pages that may be used to train AI models | You opt out of training; search is a separate setting | Your call |
| OAI-SearchBot | OpenAI | Lists pages for ChatGPT search | You are not "shown in ChatGPT search answers" | Allow |
| ChatGPT-User | OpenAI | Opens a page when a user asks | The block may not hold: "robots.txt rules may not apply" | Allow |
| ClaudeBot | Anthropic | Collects pages that could be used for training | It signals that future pages be left out of training | Your call |
| Claude-SearchBot | Anthropic | Lists pages for Claude's search | It "may reduce your site's visibility" | Allow |
| Claude-User | Anthropic | Opens a page when a user asks | Claude cannot fetch your page for that user | Allow |
| Google-Extended | A robots.txt setting for Gemini training and grounding (Gemini using your pages to answer) | Gemini may not use your pages; no effect on Google Search, says Google | Your call; not a pure training switch |
Sources: OpenAI crawler docs, Anthropic help center, Google crawler docs.
Search and user bots are how an AI assistant can name you to a buyer, so my call is to allow them. A blanket "block all AI bots" rule that covers OAI-SearchBot also removes you from ChatGPT search answers. Among sites on Cloudflare, far more block training than search.
17% vs under 1%
Sites on Cloudflare blocking AI training crawlers, against sites blocking search crawlers
Robots.txt is unlikely to stop aggressive monitors and scanners, since it relies on the bot obeying. Straus used rate limits and blocks.
What does this mean for your GTM strategy?
Traffic is not a count of buyers, and blocking AI by reflex can shut out the bots that can name you to buyers. On Cloudflare's network AI bots were a small slice of bot requests, so clean numbers come from counting humans, not a blanket AI block. That block can also drop you from ChatGPT search answers.
Before calling a launch or channel a success, check the number came from a tool that filters bots, and recompute conversion on human visits. Then set robots.txt by job, treating training access as a business decision.
Run it this week
- Pull one week of server or CDN logs and list the 20 user agents (bot name tags) sending the most requests.
- Put that week's request count next to GA4 sessions for the same days, and note the gap.
- Recompute your last launch's conversion rate on GA4 sessions instead of raw visits.
- Rewrite robots.txt by job: allow OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User, then decide on GPTBot, ClaudeBot and Google-Extended.
- Rate-limit anything hitting you many times a minute, and add a GA4 report for visits from chatgpt.com, claude.ai and perplexity.ai.
Questions people ask
What is AI bot traffic?
It is page requests from AI companies' bots and agents, not people. On Cloudflare's network, AI bots made 5.8% of web page requests in the four weeks to 14 October 2026, against 54.3% for non-AI bots.
Should I block AI bots from my website?
Block by job. OpenAI says a site that blocks OAI-SearchBot is not "shown in ChatGPT search answers", though it can still appear as a navigational link. Blocking GPTBot only opts you out of training (OpenAI).
How do I identify bot traffic in GA4?
GA4 removes known bots without showing how many. To see the rest, compare GA4 sessions with your server or CDN logs for the same days.
Does Cloudflare block AI bots?
In July 2025, Cloudflare began blocking AI training crawlers by default on new domains unless the owner opted otherwise (Cloudflare, 2026). In July 2026 it split its single "block AI bots" switch into search, agent and training controls, Free plan included (Cloudflare).
Do AI bots ignore robots.txt?
Some may. Anthropic says its bots honor it, while OpenAI says robots.txt "may not apply" to user-triggered ChatGPT-User visits. Cloudflare accused Perplexity of evading blocks in 2025 (Cloudflare), which Perplexity disputed (TechRadar).
Got this by email? Hit reply: after step 2, how far apart were your server's request count and your GA4 sessions for the same week?
Harshil
Sources
- Hacker News news.ycombinator.com
- Sammy Straus sammystraus.com
- Cloudflare, 2026 blog.cloudflare.com
- Cloudflare Radar radar.cloudflare.com
- Google Analytics Help support.google.com
- OpenAI crawler docs developers.openai.com
- Anthropic help center support.claude.com
- Google crawler docs developers.google.com
- Cloudflare, 2026 blog.cloudflare.com
- Cloudflare blog.cloudflare.com
- TechRadar techradar.com