Not Every Visitor Is Human
A substantial portion of web traffic comes from automated programs — bots. Some are useful, like search engine crawlers. Others scan pages for vulnerabilities, scrape content, or generate so much load that real visitors get slowed down.
How to Spot Harmful Bots
A single request tells you little. The pattern tells you everything:
- Hundreds or thousands of requests in a short timeframe from the same source
- Requests that pretend to be a browser but don't behave like one
- Access to paths that a normal visitor would never visit
- Sessions that appear by the second and never return
In one case, distributed bots created millions of empty sessions in a short time and flooded the session storage with gigabytes of garbage — until the site noticeably crashed. This only became visible when we looked at the load by pattern instead of by individual request.
Why Simple Blocks Aren't Enough
Blocking a single address helps little when the next bot comes from a different network. Effective protection works with patterns and thresholds: anyone behaving suspiciously gets throttled or blocked — automatically and with restraint, so real visitors and search engine crawlers get through unimpeded.
What We Set Up in Practice
- Evaluate traffic by behavior rather than by individual address
- Automatically and temporarily block suspicious sources, with a central blocklist
- Deliberately allow real crawlers so visibility doesn't suffer
- Make load and blocks visible instead of blocking blindly
Bot protection isn't a one-time wall, but an ongoing reconciliation. Done right, only those it's meant to block will notice it.