=EF=BB=BFHow Reddit M= igrated Petabyte-Scale Kafka from EC2 to Kubernetes 3D""

=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F= =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87= =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F &n= bsp; =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2= =AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F = =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD= =CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2= =80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD= =8F =E2=80=87 =C2=AD=CD=8F =E2=80=87 =C2=AD=CD=8F =E2=80= =87 =C2=AD=CD=8F =E2=80=87 =C2=AD

Forwarded this email? Subscribe here for more

How Reddit Migrated Pet= abyte-Scale Kafka from EC2 to Kubernetes

<= /tbody>

ByteByteGo

Mar 17

READ IN APP

Cut Code Review Time & Bugs in Half (Sponsored)

Code r= eviews are critical but time-consuming. CodeRabbit acts as your AI co-pilot,= providing instant Code review comments and potential impacts of every pull r= equest.

Beyond just flagging issues, CodeRabbit provides one-c= lick fix suggestions and lets you define custom code quality rules using AST= Grep patterns, catching subtle issues that traditional static analysis tool= s might miss.

CodeRabbit reviews 1 million PRs every week acr= oss 3 million repositories and is used by 100 thousand Open-source projects.=

CodeRabbit is free for all open-source repo=E2=80=99s.

Get Started T= oday
The Reddit Engineering Team completed one of the m= ost demanding infrastructure migrations in the company=E2=80=99s history. It= moved its entire Apache Kafka fleet, comprising over 500 brokers and more t= han a petabyte of live data, from Amazon EC2 virtual machines onto Kubernete= s.
The migration was done with zero downtime and without aski= ng a single client application to change how it connected to Kafka.
In this article, we will look at the breakdown of this migration, the c= hallenges the engineering team faced, and how they achieved their goal of a s= uccessful migration.
Disclaimer: This post is based on pu= blicly shared details from the Reddit Engineering Team. Please comment if yo= u notice any inaccuracies.
The Role of Kafka at Reddit
<= p style=3D"margin: 0 0 20px 0;color: rgb(54,55,55);line-height: 26px;font-si= ze: 16px;">To put things into perspective, let us first understand what exac= tly Apache Kafka is.
Apache Kafka is an open-source message s= treaming platform. Applications called producers write messages into Kafka p= artitions, and other applications called consumers read those messages out. K= afka sits in the middle and stores those messages reliably, even if the prod= ucer and consumer are running at completely different times. A single Kafka s= erver is called a broker, whereas a collection of brokers working together f= orms a cluster.
At Reddit, Ap= ache Kafka is not a peripheral tool. It sits underneath hundreds of business= -critical services, processing tens of millions of messages every second. If= Kafka went down, large portions of Reddit would break.
Why Reddi= t Wanted to Move Away from EC2
Before the migration, Reddit m= anaged its Kafka brokers on Amazon EC2 instances using a combination of Terr= aform, Puppet, and custom scripts. Operators handled upgrades, configuration= changes, and machine replacements by running commands directly from their l= aptops. This worked fine until a certain point. However, as the fleet grew, i= t became increasingly slow, error-prone, and expensive. Reddit needed a more= scalable and reliable way to operate Kafka.
Kubernetes, pair= ed with a tool called Strimzi, offered that path.
Kubernetes i= s an open-source platform for running and managing containerized application= s. Instead of manually provisioning and maintaining individual servers, Kube= rnetes lets developers describe what should be running and handles deploymen= t, scaling, and recovery automatically. Strimzi, on the other hand, is a pro= ject under the Cloud Native Computing Foundation that specifically lets you r= un Kafka on Kubernetes. It provides a declarative way to manage Kafka cluste= rs. This means that developers can describe what they want in a configuratio= n file, and Strimzi handles deployment, upgrades, and maintenance. This prom= ised fewer manual interventions and more predictable operations.
= The Four Constraints That Shaped the Migration
Reddit did no= t jump straight into moving brokers. Before writing a single line of migrati= on code, Reddit identified four hard constraints that ruled out entire categ= ories of approaches. The constraints are as follows:
Kafka ha= d to stay up. There was no acceptable maintenance window. Downtime, data los= s, or forcing client applications to change their configuration was not an o= ption. This ruled out scheduled cutovers, dual-write strategies, and replay-= based migrations.
Ka= fka=E2=80=99s metadata could not be rebuilt from scratch. Apache Kafka maint= ains a detailed internal state called metadata. This includes information ab= out which brokers exist, which broker holds which data, and where replicas o= f that data are stored. ZooKeeper, an external service, was responsible for m= anaging this metadata. There is no supported way to recreate this metadata o= n a fresh cluster while keeping the system available. New brokers had to joi= n the existing cluster rather than replace it.
Client connectivity was tightly coupled to specifi= c brokers. Over time, applications across Reddit had been configured to conn= ect directly to specific broker hostnames, typically the first few brokers i= n a cluster, rather than using a single load-balanced endpoint. Turning off t= hose brokers would immediately break hundreds of services. Reddit did not co= ntrol the layer through which clients found and connected to Kafka.
=
Every step had to be reversi= ble. No single action during the migration could leave the system in a state= from which recovery was impossible. This meant Reddit had to accept a long p= eriod where EC2 brokers and Kubernetes brokers ran side by side, and it mean= t that riskier changes had to wait until everything else was stable.
Phase 1: Taking Control of the Naming Layer
Th= e first phase of the migration did not touch Kafka at all.
Re= ddit introduced a DNS facade, which is a set of DNS records that act as an i= ntermediate layer between client applications and the actual Kafka brokers. D= NS is the system that translates human-readable names into the addresses of s= ervers. By creating new, infrastructure-controlled DNS names that initially p= ointed to the same EC2 brokers, Reddit changed nothing from the perspective o= f client applications.
Reddit then rolled out these new conne= ction strings across more than 250 services using automated tooling that gen= erated batch pull requests to update configuration files. Once all clients w= ere talking through this DNS layer, Reddit could change where those names po= inted, from EC2 to Kubernetes, without modifying any client code.
Phase 2: Making Room for New Brokers
Each Kafka broker is i= dentified by a unique numeric ID. Strimzi assigns broker IDs starting at 0 b= y default. However, Reddit=E2=80=99s existing EC2 brokers already occupied t= hose low numbers.
To free up that ID space, Reddit doubled th= e cluster size by adding new EC2 brokers with higher IDs, then terminated th= e original low-numbered brokers. This shifted all data onto the higher-numbe= red brokers and opened up IDs 0, 1, 2, and so on for Strimzi-managed brokers= to use.
See the diagram below:
Phase 3: Running a Mixed Cluster
This was t= he most technically complex phase.
Reddit needed Strimzi broke= rs running on Kubernetes to join the same cluster as the existing EC2 broker= s and communicate with them directly. Strimzi does not support this out of t= he box, so Reddit created a fork of the Strimzi operator. The changes Reddit= made were deliberately small and targeted:
The inter-broker l= istener configuration was set to use plaintext listeners accessible from bot= h EC2 and Kubernetes, ensuring brokers in different environments could talk t= o each other.
The Zoo= Keeper connection was pointed at Reddit=E2=80=99s existing EC2-hosted ZooKee= per, so that both old and new brokers shared the same metadata store and wer= e part of the same logical cluster.
The Cruise Control topic was overridden to stay consistent a= cross both broker sets, allowing Reddit to use Cruise Control to move data b= etween EC2 and Kubernetes brokers. Cruise Control is a Kafka tool that autom= ates the process of rebalancing data across brokers in a controlled, measure= d way. It was central to the actual movement of data during the migration.
Running a forked oper= ator in production carries risk. Reddit kept the scope of changes narrow and= planned from the start to switch back to the standard Strimzi operator once= the migration was complete.

<= /td>
Phase 4: Gradually Shifting Data and Traffic
With= both sets of brokers running inside the same cluster, Reddit used Cruise Co= ntrol to incrementally move partition leadership and replicated data from EC= 2 brokers to the Kubernetes brokers.
Partition leadership det= ermines which broker is responsible for serving reads and writes for a given= piece of data. Kafka stores copies of each partition on multiple brokers fo= r redundancy. This is called the replication factor. Moving data meant reass= igning both the leadership and the replicas to the new set of brokers, one p= artition at a time.
Reddit monitored this process continuousl= y as the partition leadership on EC2 declined steadily over roughly a week w= hile leadership on Strimzi climbed in parallel. Network traffic followed the= same pattern. At every point, Reddit could pause or reverse the process if s= omething looked wrong.
See the dashboard view below:

3D"" — **<= strong>Source:** Reddit Eng= ineering Blog

Phase 5: Migrating the C= ontrol Plane

ZooKeeper had managed Kafka=E2=80=99s metadata t= hroughout the entire broker migration. Reddit made a deliberate choice not t= o change the control plane until after the data plane was fully stable on Ku= bernetes. This separation of concerns reduced the risk of compounding failur= es.

Once all EC2 brokers were terminated and all data and tra= ffic were running on Kubernetes, Reddit executed the migration from ZooKeepe= r to KRaft. KRaft is Kafka=E2=80=99s built-in metadata management system tha= t eliminates the need for ZooKeeper.

See the diagram below:

Since Strimzi and Kafka bot= h provide documented steps for this migration, and because the rest of the s= ystem had already settled, this final phase was comparatively straightforwar= d.

Phase 6: Cleaning Up and Handing Off to Standard Strimzi

<= p style=3D"margin: 0 0 20px 0;color: rgb(54,55,55);line-height: 26px;font-si= ze: 16px;">After both the data plane and the control plane were fully runnin= g on Kubernetes, Reddit removed all the configuration overrides that the for= ked Strimzi operator had introduced.

Control of the clusters w= as handed off to the standard, unmodified Strimzi operator. The EC2 infrastr= ucture was decommissioned.

Conclusion

Reddit=E2=80= =99s migration is a good example of how large-scale infrastructure changes d= o not have to be dramatic, high-risk events. By breaking the work into small= , reversible, well-understood steps and by respecting the constraints the sy= stem imposed, Reddit moved a petabyte-scale platform to Kubernetes without a= single moment of downtime.

Some key lessons from Reddit=E2=80= =99s migration journey were as follows:

Introducing a control= lable abstraction layer between clients and infrastructure, whether that is D= NS, a proxy, or an API gateway, is one of the highest-leverage changes you c= an make during a migration. It decouples the two sides and lets you change t= he infrastructure without forcing every team to update their code.

Metadata and logical state te= nd to outlive the physical machines they run on. When planning any large mig= ration, treat the logical state as the thing you are protecting, and treat t= he infrastructure as something you are replacing around it.

Designing each step to be undoable i= s not just a safety measure. It changes how confidently and quickly you can m= ove forward, because you know you can always step back if something goes wro= ng.
<= p style=3D"color: rgb(54,55,55);line-height: 26px;margin-bottom: 0;box-sizin= g: border-box;padding-left: 4px;font-size: 16px;margin: 0;">A migration that= looks messy in the middle but never breaks production is far preferable to a= clean design that requires a moment where things could go wrong with no rec= overy path.

References:<= /strong>

Swapping the Engine Mid-Flight: How We M= oved Reddit=E2=80=99s Petabyte Scale Kafka Fleet to Kubernetes

<= li style=3D"margin: 8px 0 0 32px;mso-special-format: bullet;">
What is Strimzi
KRaft: Apache Kafka without Zookeeper=

<= /tbody>

Comment

Restack