The Brew
Ha Ha
Cover storyHosting V4
Half the crowd
isn't human
What agent traffic does to a single-server stack
A person shopping for a jacket opens a handful of pages. An agent running the same errand opens hundreds.
7 min read
Also in this issue
- The cache you know as Varnish renamed itself
- 23,951 pages crawled per referral sent back
- An eight-step checklist before you upgrade
Contents
Six departments, one platform release and a rename that will break a deploy if you skip it.
- 01The first pourYour cache notices before your dashboard doesp. 01
- 02Order upShip log, every item traced to a dated pagep. 04
- 03Under the hoodDefaults, boundaries and one renamep. 06
- 04Bean counterFigures on the traffic shift, with sourcing on eachp. 09
- 05From the blogManaged hosting vs infrastructure-layer commercep. 11
- 06The grindPost-release checklist, eight stepsp. 12
01 / The first pourIssue 07
Your cache notices before your dashboard does
Shopping sites are the most crawled category on the web right now, and those requests arrive with cookies and session patterns that look human. They land on your origin like real users.
Cloudflare's own prediction put the crossover at the end of 2027. It landed eighteen months early. Automated requests are now the majority of HTML traffic on the web, and Fastly measured AI requests growing several times faster than human traffic through the first half of this year.
That load profile is a caching question before it becomes a capacity question. Vinyl sits in front of the web tier and absorbs the repeat reads, which is most of what an agent generates. What happens to everything the cache misses depends on how much of your stack shares one machine.
The request pattern changed faster than the forecasts
V4 breaks that sharing apart in documented steps. Standard lifts the web tier into a layer that scales on request demand, leaving a core server for databases, caching, search, NFS and cron. Pro moves MySQL to RDS, Valkey to ElastiCache and OpenSearch to the AWS service, so the core server holds only shared files, cron and message queueing. Pro HA mirrors most of that across two availability zones with automatic failover.
The tier you run decides how much of your stack a traffic surge can reach.
What this does not do
The tiers give you headroom. They don't tell you which requests came from an agent, so classification stays your job, and Web Controls is where you act on it. V4 also documents no migration path from V3. If your environments run V3 today, changing platform is a support conversation and a plan, not a toggle in the Control Panel.
Your cache notices before your dashboard does.
02 / Order upIssue 07
Order up
Ship log
Every item traces to a dated page on docs.webscale.com.
Web controls
Traffic conditions can match on ASN
Two conditions landed on July 10, ASN is and ASN is not. Automated traffic arrives from identifiable cloud autonomous systems, so shaping by ASN beats chasing IP ranges that rotate underneath you.
Vinyl
The cache you know as Varnish renamed itself
The open source project became Vinyl Cache after a trademark dispute, with 9.0.0 landing in March. Varnish Software ships a separate downstream distribution that keeps the old name. V4 documents the upstream one.
Hosting V4
The platform ships in documented tiers
Essentials, Standard, Pro and Pro HA each carry an architecture overview and a plan specification. Magento, Shopware, WordPress and custom PHP or Node.js applications are all in scope. Documented July 5.
Valkey
Caching ships as two services
The V4 catalog documents valkey-volatile and valkey-persistent separately. Pick per workload. A session store that needs to survive a restart wants the persistent one.
Commands
The V4 command set is documented
dev, crontab-edit, build-php-image, rabbitmqctl and supervisorctl. If your deploy scripts call V3 equivalents, they need a read before you move.
Image catalog
Release 2026.34 is live
Landed July 22. Check your tags against it.
docs.webscale.com/latest-updates →03 / Under the hoodIssue 07
Defaults, boundaries and one rename
Notes on the plumbing, starting with the one that will break a deploy if you upgrade without reading it.
Caching
Vinyl, and what the rename breaks
The rename came out of a trademark dispute, and it isn't cosmetic. There's no compatibility layer. Arch Linux dropped the varnish package outright in late May and replaced it with vinyl-cache.
Everything moved. Config goes from /etc/varnish to /etc/vinyl-cache. State goes from /var/lib/varnish to /var/lib/vinyl-cache, and file ownership needs fixing after. The varnish user and group become vinyl, and varnishlog becomes vinyllog. The systemd units change from varnish.service and varnishncsa.service to vinyl-cache.service and vinylncsa.service. There are breaking VCL changes on top of that.
What trips people is that the binary is still varnishd and still reports a varnish version string. Only the copyright line tells you which project you're on. The 7.7 series is unsupported now, and VSV00018 was patched in 9.0.0.
Configuration
The documented Vinyl defaults
vinyl.cache_size = 1G
Setting the pool size turns the cache on. The V4 docs publish the rest of the baseline, so you can diff a config you inherited rather than guess at it.
Tier boundaries
What the core server stops holding
The interesting part of the tier ladder is subtraction. Essentials keeps everything on one box. Standard keeps databases, caching, search, NFS and cron on the core while the web tier scales away from it. Pro strips the core down to shared files, cron and message queueing, because RDS, ElastiCache and the AWS OpenSearch service take the rest.
Read the tiers as a list of what leaves.
Dig into the developer docs →Valkey
You're probably already running it
Valkey is the Linux Foundation fork of Redis 7.2, made after the 2024 license change. Two years on it's the default for new ElastiCache and MemoryDB, distributions switched their defaults, and the Redis binary in a stock container image is usually Valkey now. V4 landing on it follows the industry rather than leading it.
The thing worth checking is modules. RediSearch, RedisJSON and RedisTimeSeries are where compatibility gets uneven, and that's the common snag in a Magento setup.
04 / Bean counterIssue 07
Bean counter
Figures on the traffic shift, with sourcing on each
Three figures / two tiers of confidence
57.5%
Share of HTML web traffic coming from automated requests, against 42.5% from humans. Cloudflare's own public forecast had put this crossover at the end of 2027.
Cloudflare Radar, 2026. Vendor telemetry, roughly a fifth of all websites. Tier 1.
6.5x
How much faster AI requests grew than human traffic on Fastly's network from January through May 2026, with AI requests up about 30% over that window.
Fastly, June 2026. Vendor telemetry. Tier 1.
23,951:1
Pages crawled per referral sent back, for one major AI crawler. Traditional search runs closer to five to one. Most AI crawling is extraction rather than discovery, which is why it shows up as origin load and not as sessions.
Third-party analysis of Cloudflare data, 2026. Tier 2.
05 / From the blogIssue 07
Managed hosting vs infrastructure-layer commerce
Adrian's take on the infrastructure-layer question from the merchant side, worth reading next to the tier ladder because both turn on who controls what underneath the storefront.
One reports the fire. The other is plumbed in to put it out.
06 / The grindIssue 07
The grind
Post-release checklist
Work through this in one sitting.
- 01Check whether you're running Varnish or Vinyl, and which version
- 02Back up
/etc/varnishbefore any package upgrade touches it - 03Map the renamed paths, users and systemd units before you upgrade
- 04Audit your VCL against the 9.x breaking changes
- 05Diff your cache parameters against the V4 documented defaults
- 06Pull your ASN distribution from traffic logs, then add Web Controls conditions for the sources you want to shape
- 07Confirm which V4 tier each environment runs today
- 08Check Valkey module dependencies if you rely on RediSearch or RedisJSON