99% of My Website Traffic Is Bots

A year of fighting scrapers on my 1.5 million-page website. What I tried, how I failed, and what's working now.

If you're trying to steal my data: start here! I'll tell you exactly what firewall rules I have in place so you can work around them more easily.

I'm sort of joking. I don't even know if a web crawler reads blog posts to get tips like this.

But I've spent a year fighting bots on my philanthropy donor database, PatronView. Nobody teaches you this stuff when you're vibe coding, so I'll share what I've learned.

Some of what's below:

My site got 3.6 million requestsfrom Chinese bots in just one day.
Cloudflare says Anthropic's crawlers run about 3,000 crawls per visitor referred. I measured mine at 35,000:1.
Two days ago I blocked Amazon's AI search crawler. It was reading 117,000 pages a dayand has never sent me a single visitor.
My CAPTCHA solve rate is 0.24%. The bots don't even try.
And every firewall rule I run is listed at the bottom.

214 bot page loads for every 1 human

In the week I published this post, my server answered 2.5 million requests from the outside world and served 1.28 million full pages. But my visitor stats only recorded 5,977 pageviews.

So for every page load I see, about 214 more happen that I never see.

That's where the title comes from. 5,977 human pageviews out of 1.28 million pages served is less than half of one percent. If anything, calling it 99% bots is me rounding down.

My visitor stats show a tidy little site, about 500 visitors a day. But my server answers millions of real requests every week.

If you only look at visitor stats from a JavaScript tool like Plausible or Fathom or Google Analytics, you have no idea what's hitting your server. I didn't.

The first botnet

PatronView is a custom database of American philanthropists that I cobbled together with various levels of intelligence. It has 1.5 million individual profile pages built from IRS 990 forms, public donor walls, and annual reports.

And yes, my site gets its data by scraping those public documents. So I'm a scraper writing a blog post complaining about scrapers. I'm aware of how that sounds. For the record, I scrape my sources a few times a year, not thousands of times a day.

From the beginning, SEO crawlers like SemrushBot, AhrefsBot, MJ12bot, and DataForSEOBot pounded the site, crawling every single page over and over. Blocking them in robots.txt and then at the Cloudflare level with security rules was my gateway drug into all of this.

Then in November 2025, four thousand "visitors" showed up over a few days. Each visited exactly one page with a bounce rate of 99%. More telling was that they had no referrer, which is usually the easiest way I spot a bot. And they were only crawling my fund pages (like this one, this one, and this one), which only 10% of real visitors ever touch. But those 4,000 bots were just the warm-up.

The day China arrived

On April 22, my site took 3.6 million requests in a single day. They came from 361,844 unique IP addresses, and nearly all of them were in China.

Cloudflare's Managed Challenge (their invisible CAPTCHA) absorbed 1.18 million of those in the first ten hours. And yet an alarming number of these Chinese bots were passing the challenge.

So I did the thing you're not supposed to do: I blocked the entire country of China at the edge. A few days later, a copycat wave from Vietnam started, so I blocked Vietnam, too. And then Singapore.

My site is a database of American museum donors, in English. My real search traffic is 95.9% United States and 1.3% Canada.

When I tweeted about the flood of Chinese web traffic, the replies told me everyone is fighting this:

Matt Paulsonfrom MarketBeat: "Don't forget to add Russia to the list."
Jack Ellisfrom Fathom Analytics: "We saw a ton of spam traffic from China over the last 6 months. I will say that customers have seen China drop completely now."
Jeremy Brandt: "Our country block list on Cloudflare for all of our sites is... extensive."
Chris Lewicki: "I decided to stop fighting it. My counter-measures were introducing too much complexity in my life."
Rodrigo Rocco: "It's been crazy lately, they are using thousands of residential IPs making only 1 call each, so hard to stop it."

Remember that last one from Rodrigo, because it comes back to bite me later.

The Claude ratio

Cloudflare has said Anthropic's crawlers run at about 3,000 pages crawled per visitor referred. In June, I measured mine: 35,000 to 1.

To be clear about what that means: for every one visitor Claude sent me, its crawler read 35,000 of my pages.

I found it in Cloudflare's AI crawler dashboard, shown below. Claude-SearchBot, Anthropic's search crawler, had requested 420,680 pages in one week. That same week, Claude sent me 12 human visitors, measured by Claude-User, the separate user-agent Anthropic sends when a real person asks Claude to fetch a page. That's the flat line at the bottom.

The bandwidth tab was worse. That week I served 4.63 GB to the bot, and 175 KB to the humans it sent me:

I love Anthropic and Claude as much as the next guy. My website was built with Claude Code, and I used Claude Code to help me refine my security rules on the Cloudflare API. Complaining about Claude scraping the site that I used Claude to build is a level of irony I've made peace with.

But Anthropic and Claude do not send me traffic. So I blocked Claude-SearchBot at the firewall.

And they respected it! The crawler dropped from 60,000 requests a day to about 25 attempts a day. The polite AI companies really do take the 403. (A 403 is the web's "access denied" response.)

Pages crawled per visitor referred

That block gave me the metric I now use for everything: pages crawled per visitor referred. Google earns its keep. Bing is 9x worse but still defensible. The AI crawlers that are training LLMs are not close:

So two days ago I blocked it the same way. It took two minutes.

Cloudflare's managed AI Crawl Control does a lot of heavy lifting here too. It blocks the declared training bots (GPTBot, ClaudeBot, CCBot, Bytespider) before my rules even run.

That same dashboard shows Bingbot at 158,610 requests for 680 visitors. My Bing referrals are actually growing, so I'm happy for them to keep scraping. And it shows Anthropic down to 47 requests since my block. (Click here to see a screenshot of the crawler dashboard.)

The American datacenter wave

In July, someone new came knocking or else those scrapers adapted to my geography rules. They came from the United States instead.

A steady wave of headless Chrome browsers on AWS. These ran JavaScript and polluted my stats with thousands of fake "visitors."

The fix was the most durable rule I've built: challenge every datacenter.

Real readers browse from Comcast and T-Mobile, and almost never from an AWS us-east-1 IP address. The exceptions are people on cloud desktops or VPNs, which is why this rule is a challenge instead of a block.

The day I turned off the CAPTCHA

Cloudflare's "JavaScript Detections" feature had been injecting a challenge script into every page, all year. I guess I turned it on at some point? And maybe it was supposed to be helping? But then I found it this week while trying to improve my site speed scores.

That script cost 2,875 milliseconds on a mid-range phone. My entire site's own JavaScript runs in 278ms. It was the single biggest reason my mobile Lighthouse score was 58, and on my Cloudflare plan its verdict isn't even readable by firewall rules. So I was taking a 40-point performance tax for telemetry nobody could read.

On August 5, I turned it off and hit a Lighthouse score of 99 within an hour.

Five hours later, a scraper on Microsoft Azure took 23,000 pages in one hour, from more than 80 IP addresses, each one politely staying under my rate limit. So I guess the JavaScript Detections was working. But then I turned it off, and needed a new plan.

And a botnet of residential IPs showed up pretending to be Chrome 118 through 120. Browser versions from 2023, frozen in whatever scraping toolkit they downloaded.

The residential botnets

A residential botnet is scraping traffic routed through thousands of ordinary home internet connections, rented by the request from proxy companies. Every request looks like a real person on a real ISP.

I'd seen this shape before. Back in November 2025, I noticed something strange in my visitor stats: dozens of countries visiting my site in almost identical numbers on the same day. Here's the screenshot from my analytics:

That even spread across forty countries is the signature of a rented proxy network: thousands of real-looking connections, all working for the same customer. It's exactly what Rodrigo warned about in those tweet replies.

Nine months later, the same shape came back, this time on American home internet connections. Both of my best rules sort traffic by where it comes from. One challenges anything outside North America. The other challenges anything from a big cloud provider. This traffic is neither. It arrives from a house in Ohio on Spectrum, so both rules wave it through. You can see it ramp up in this chart of unique IP addresses hitting my site each day in July:

I responded with two new rules. First, Azure and the other big clouds joined the datacenter challenge list.

And I added my favorite dumb rule: challenge browsers frozen years in the past. Chrome 100 through 130 gets a CAPTCHA. So does old Firefox. I checked my real traffic first: only 0.54% of my actual search visitors run browsers that old, and most of those are on Firefox 115 ESR, which I exempted. The frozen botnet fails this rule all day long.

My Cloudflare Security rules

Everything runs on Cloudflare's WAF. I pay for the Pro plan at $25 per month. The logic ports to any modern firewall. Here are the rules I'm running today:

Block China and Vietnam.See above. Country blocks add zero latency for everyone else.
Block 12 SEO crawlers by user-agent.Semrush, Ahrefs, MJ12bot, DotBot, BLEXBot, Barkrowler, and friends. They identify themselves honestly, bless them.
Block AI crawlers with bad ratios by user-agent.Right now:- Claude-SearchBotand- Amzn-SearchBot. They can still read- robots.txt, so they know they're not welcome. My- robots.txtseparately tells- GPTBot,- ClaudeBot,- CCBot,- Bytespider, and the other declared training bots to go away. The polite ones comply.
Skip everything below for verified bots.Cloudflare cryptographically verifies Googlebot, Bingbot, and Applebot. The ordering is the trick: this skip sits after my block rules. A "verified" bot I've blocked stays blocked, but no challenge ever touches Google.
Challenge every continent except North America.The blunt one, and the one I'd defend hardest. My audience is 97% North America. Everyone else gets one invisible CAPTCHA per 45 minutes.
Challenge empty user-agents.No real human's browser sends an empty user-agent string.
Challenge 46 datacenter ASNs.Humans don't browse from AWS. (Generally.)
Challenge stale browsers.The frozen botnet rule.
Rate limit: 30 page requests per 10 seconds per IP.Static files excluded, so it never fires on a normal page load.

My robots.txt also blocks all 12 SEO crawlers by name, and Cloudflare adds its own AI-training blocks on top of it at the edge. You can read the live file at patronview.com/robots.txt.

Does all this challenging hurt real humans? Barely.

Over a recent 48 hour window, Cloudflare issued 106,437 challenges. 252 were solved. That's a 0.24% solve rate.

My server costs

The site runs on Cloudflare Workers with the D1 database and the KV cache at the edge. Most bot requests hit cache and cost fractions of a penny. My normal bill for running this whole site is around $90 a month. During one bad spike month, it jumped about 500%.

The bots use 99% of the bill and I pay 100% of it.

The real costs were the sneaky ones:

My visitor stats got so polluted I couldn't trust my own numbers.This was the one that hurt. I'm trying to run a business here. I want to know what real people read on my site so I know what to build next. I couldn't see them through the bots.
One afternoon, cache misses from the Azure burst piled up and real visitors got 504 errors.

If I were on a traditional VPS paying for CPU and bandwidth, millions of weekly requests where over 99% are bots would be an existential problem. On an edge platform, it's a nuisance.

Some of that bill is my own fault, to be clear. I'm still cleaning up inefficient queries and cache settings that made those spike months worse. I keep thinking about switching to a VPS and hosting my own local database, but I've resisted because I like playing with the Cloudflare platform.

I think that asymmetry is why scraping keeps getting worse. The scrapers' costs fell faster than everyone's defenses improved.

What to do if this is happening to you

Look at your server logs, not your visitor stats.Your stats see less than half a percent of what's happening.
Block by ASN, not by IP.IPs rotate forever. Networks don't.
Challenge, don't block, when humans might be inside.A managed challenge is invisible to most real people. And it lets you measure.
Watch your solve rates.A 0.2% solve rate means bots, so keep the rule. A 30% solve rate means you're taxing humans, so fix the rule.
Profile what your security features cost.My CAPTCHA script was more expensive than my entire site.

Conclusion

So did any of it work?

In the 24 hours since I finished these rules, Cloudflare blocked 46,729 requests outright. 43,150 of those were Amazon's crawler alone, still hammering the block I set up two days ago.

It also issued 63,969 challenges, and only 552 of them were solved.

So the rules are working. Hopefully they keep letting through the bots I actually want, and the humans. At least for now.

Let me tell you, this feels like whack-a-mole. Every few weeks I have to spend time fighting somebody new away.

There's a real conflict here. I want Google and Bing and DuckDuckGo to crawl my site and send me new readers. But I don't want everyone else strip-mining it. And the residential botnets already solve CAPTCHAs and look exactly like real browsers. Blocking them at the network level is impossible by design.

Scraping keeps getting worse because it keeps getting cheaper, so I think the only real fix is economic. Cloudflare is building pay-per-crawl, where crawlers pay per request at the edge. I'd happily sell Amazon those 3.5 million pages a month at a fair rate.

Until a market like that exists, my rule is simple: a crawler that never sends me a visitor gets blocked.

If you're fighting this too, email me: hello@nickgray.net. I'd love to compare notes.

Appendix: the exact rule expressions

For the technical crowd. These are the actual expressions behind the rules above, copied out of my Cloudflare zone via the API. They're written in Cloudflare's rules language and should port to most firewalls with light translation. I've omitted a couple of internal housekeeping skips.

Block China and Vietnam. Two separate rules, action: Block.

(ip.src.country in {"CN"}) (ip.src.country eq "VN")
Block SEO crawlers by user-agent. Action: Block.

(lower(http.user_agent) contains "barkrowler") or (lower(http.user_agent) contains "thinkbot") or (lower(http.user_agent) contains "brightbot") or (lower(http.user_agent) contains "mj12bot") or (lower(http.user_agent) contains "semrushbot") or (lower(http.user_agent) contains "siteauditbot") or (lower(http.user_agent) contains "ahrefsbot") or (lower(http.user_agent) contains "ahrefssiteaudit") or (lower(http.user_agent) contains "dataforseobot") or (lower(http.user_agent) contains "dotbot") or (lower(http.user_agent) contains "blexbot") or (lower(http.user_agent) contains "splitsignalbot")
Block AI crawlers with bad ratios. Action: Block. Note the carve-out: a blocked crawler can still read robots.txt, the file that tells it why it's blocked.

(http.request.uri.path ne "/robots.txt" and (http.user_agent contains "Claude-SearchBot" or http.user_agent contains "Amzn-SearchBot"))
Skip everything below for verified bots. Action: Skip (remaining custom rules + rate limiting). One expression. Order matters here: it sits after the blocks above, so a blocked bot stays blocked, but Googlebot never sees a challenge.

(cf.client.bot)Challenge every continent except North America. Action: Managed Challenge. The country carve-out at the end is because Guam, American Samoa, and the Northern Mariana Islands are US territories that carry Oceania continent codes. Real American readers live there. T1 is the Tor network.

((ip.src.asnum in {212238 139341 9009}) or (ip.src.continent in {"AF" "AN" "AS" "OC" "SA" "T1" "EU"})) and not cf.client.bot and not (ip.src.country in {"GU" "AS" "MP"})
Challenge empty user-agents. Action: Managed Challenge.

(http.user_agent eq "") and not cf.client.botChallenge datacenter and cloud ASNs. Action: Managed Challenge. AWS, Azure, Google Cloud, Oracle, Alibaba, DigitalOcean, Vultr, Linode, OVH, Hetzner, Hurricane Electric, and every hosting network I've caught scraping.

(ip.src.asnum in {14618 16509 8987 396982 14061 20473 63949 16276 24940 213230 55286 64286 36352 18779 40676 397423 46261 399073 2914 13332 30058 11798 21769 62874 202015 55933 393886 27411 396362 395954 396190 19148 30633 8075 8070 8068 8069 31898 45102 37963 36351 136907 45090 207990 17497 6939}) and not cf.client.bot
Challenge stale browser majors. Action: Managed Challenge. Cloudflare's Pro plan has no regex matching in rule expressions (that's a Business feature), so yes, this is 55 contains clauses. Firefox 115 is skipped because it's the old ESR release my real visitors still run, stranded on older operating systems. It needs a bump every quarter as versions age.

Expand the full expression

(http.user_agent contains "Chrome/100." or http.user_agent contains "Chrome/101." or http.user_agent contains "Chrome/102." or http.user_agent contains "Chrome/103." or http.user_agent contains "Chrome/104." or http.user_agent contains "Chrome/105." or http.user_agent contains "Chrome/106." or http.user_agent contains "Chrome/107." or http.user_agent contains "Chrome/108." or http.user_agent contains "Chrome/109." or http.user_agent contains "Chrome/110." or http.user_agent contains "Chrome/111." or http.user_agent contains "Chrome/112." or http.user_agent contains "Chrome/113." or http.user_agent contains "Chrome/114." or http.user_agent contains "Chrome/115." or http.user_agent contains "Chrome/116." or http.user_agent contains "Chrome/117." or http.user_agent contains "Chrome/118." or http.user_agent contains "Chrome/119." or http.user_agent contains "Chrome/120." or http.user_agent contains "Chrome/121." or http.user_agent contains "Chrome/122." or http.user_agent contains "Chrome/123." or http.user_agent contains "Chrome/124." or http.user_agent contains "Chrome/125." or http.user_agent contains "Chrome/126." or http.user_agent contains "Chrome/127." or http.user_agent contains "Chrome/128." or http.user_agent contains "Chrome/129." or http.user_agent contains "Chrome/130." or http.user_agent contains "Firefox/100." or http.user_agent contains "Firefox/101." or http.user_agent contains "Firefox/102." or http.user_agent contains "Firefox/103." or http.user_agent contains "Firefox/104." or http.user_agent contains "Firefox/105." or http.user_agent contains "Firefox/106." or http.user_agent contains "Firefox/107." or http.user_agent contains "Firefox/108." or http.user_agent contains "Firefox/109." or http.user_agent contains "Firefox/110." or http.user_agent contains "Firefox/111." or http.user_agent contains "Firefox/112." or http.user_agent contains "Firefox/113." or http.user_agent contains "Firefox/114." or http.user_agent contains "Firefox/116." or http.user_agent contains "Firefox/117." or http.user_agent contains "Firefox/118." or http.user_agent contains "Firefox/119." or http.user_agent contains "Firefox/120." or http.user_agent contains "Firefox/121." or http.user_agent contains "Firefox/122." or http.user_agent contains "Firefox/123." or http.user_agent contains "Firefox/124.") and not cf.client.bot
Rate limit. Action: Managed Challenge when an IP exceeds 30 matching requests in 10 seconds. It only counts extensionless paths, so the 40 images and stylesheets on a normal page load never trip it. (Strictly, the counter is per IP per Cloudflare data center, so a distributed botnet can slide under it. It catches the lazy ones.)

(http.request.uri.path.extension eq "") and not cf.client.botAnd one bonus rule I keep disabled: the panic button. Same expression as the rate limit, but as a plain Managed Challenge on every page request from anything that isn't a verified bot. When a scrape burst is actively hammering the site, I flip it on, let the flood die, and flip it off within the hour.

Newspaper

99% of my website traffic is bots