The engineering team at Zalando recently described the design and implementation of an in-process, client-side load balancer for a high-throughput API handling around 1 million requests per second. The result was more predictable latency, a drop in infrastructure costs, and better visibility into where failures actually originate.

Zalando's Product Read API is designed to serve millions of requests per second with single-digit-millisecond latency across the 25 markets of one of the largest European online fashion retailers. Its batch endpoint fans a single request into up to 100 parallel calls to individual product pods, each transiting Skipper, the shared cluster edge load balancer. As a batch waited on the slowest of 100 hops through infrastructure the team didn't own, they could not separate Skipper latency spikes from their own. To address that, they moved routing for high fan-out internal traffic in-process, while keeping Skipper for edge and single-GET traffic. Conor Gallagher, senior principal engineer at Zalando, explains:

We decided that for high fan-out internal traffic, the routing decision should live inside the calling process itself. Edge traffic, where Skipper excels, should stay exactly where it is. We were not replacing Skipper; we were graduating the internal fan-out path to a client-side load balancer that runs inside the process.

Product-sets routing directly to product pods. Source: Zalando blog

The team reimplemented Skipper's exact algorithm (xxHash64, 100 virtual nodes per endpoint) so both paths route identical keys to identical pods during migration, avoiding cache splits, with unit tests pinning this behavior. Gallagher writes:

This means that adding or removing an endpoint remaps only about 1/N of keys, minimising cache churn. And because both Skipper and our library use the same hash function and the same number of virtual nodes, they produce identical rings for the same set of pods.

The article covers the engineering decisions and challenges of client-side routing, including consistent hashing, occupancy-based load balancing, cache-aware scaling, rollout strategy, observability improvements, and experiments with availability-zone routing.

The team replaced control-plane-crashing polling with a watch-based Kubernetes informer, rebuilt a pipeline that took hours into fast, reversible deploys, and ramped the load balancer from 1% to 100% behind toggles, shrinking Skipper's fleet from 50+ pods to 8 and cutting deployment daily cost from $450 to $110. To eliminate scale-up latency spikes, they added N-ring fade-in over 30 seconds on a ^2.5 curve so new pods warm only the products they'll actually serve. Werner Vogels, CTO of Amazon, writes:

I am not a big fan of doing this client-side, for a whole slew of reasons, but this is impressive engineering by the Zalando team. I like the "lessons learned" they conclude with.

Alexey Kuznetsov, principal engineer at AWS, comments:

Client side load balancing was kind of a revelation for me when I moved from Amazon to Google. (...) I think this thinking needs to be more prominent.

While Zalando’s attempt at AZ-aware routing was meant to cut inter-AZ costs, it fragmented caches and triggered DynamoDB read explosions. With the new approach, they hardened the path with a single jittered retry, FIFO overload shedding, and richer logging that exposed brief node-level freezes the balancer now routes around on its own.

Highlighting that Zalando built its own client-side load balancer only because they sat at a genuine extreme edge case, Gallagher concludes:

Should you build your own? For almost everyone, the answer is don't. A mature proxy like Skipper or Envoy gives you consistent-hash routing out of the box, maintained by someone else and hardened by thousands of other users.

On Hacker News, a user questions the choice to develop an independent LB module:

The approaches described in the article are clever and the results are impressive, so kudos to the team! However, while reading, it felt weird that they treated Skipper as a third-party dependency, even though it is developed and operated by their colleagues. I mean, they could've tried contributing some updates and improvements to Skipper, maybe reusing Skipper's discovery mechanism.

While Skipper is open source and available on GitHub, the new client-side load balancer is not published and remains internal to Zalando.