Delivered-To: tjphuhs@gmail.com Received: by 2002:a5d:51c9:0:b0:451:8158:aa77 with SMTP id n9csp516130wrv; Wed, 13 May 2026 08:41:38 -0700 (PDT) X-Received: by 2002:a05:6a00:ad0b:b0:837:e9cc:d474 with SMTP id d2e1a72fcca58-83f03e87948mr4444537b3a.2.1778686897849; Wed, 13 May 2026 08:41:37 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1778686897; cv=none; d=google.com; s=arc-20240605; b=XhdtP4/R1l3P+mCaSRMVUq6fCftDgUXlU3HhGJdLC2GrtyIVllSYH7SAr0ZjRsEU7D 62j47WlGbUhopaDKctu7gOKC/HBoManhEqcpm97AunlgUv/lptJDd4g9FhpCJg8bkjgB CGyVI4EjmMw7LQBli3YWDZ2w99LVsuwamCbH6z96YPYMo1XIqBsJhgmUBrERCIBb01hU rS2XTWddN7POV/GqeoW/tGbGIg4tTOerhle9JFlICUdV2XAlwy4ROziZb7oZ/lBvgRns wyCRnAeEGmkNyvMolD/LMvcmASvXPLGPgTZj/zJj2b3FcALL945xUVwgmXwabYZKRJrZ 3yIg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=list-unsubscribe-post:list-unsubscribe:list-post:list-id :list-archive:list-url:list-owner:reply-to:in-reply-to:references :sender:feedback-id:date:message-id:to:from:subject:mime-version :dkim-signature:dkim-signature; bh=LqUbMsWPbVOOcUkZuxru0itlB+kmYh6TpBY01tjBMws=; fh=9zP21OW67FFZD2VV42mGudo540Qp4aalMWxJB5kMYug=; b=O1qlX8bFETCE4v021uOqvG6RRIGshdzzu5PPjAjDy87Z+KwVc3095RtXCE1riYu4a1 fnLOg+3jCtw6X5FKhd/uTCvee2EiHkGENa8zS9Cni0n8vv3jZg8CgOgIlKhSDuYQLfpr Qg5wy+4tYXf8Wsyga/gVgcJaLhquXBjRAu5QB4r2G3D0J0DyDRyiZg25uIGung91VtNG I7nje0Qwx0IK5/WOhV0UjbDILhFqyuSUvrgM8QpcAt2vCVmBkTscOdI9Uu+HjWsk1OWA rYQ/kzunXHLWbju19aoLEka1ervrsKwmTlIXfAJayUEmrbji44I1iyuaIggs01U0VBJn yvpg==; dara=google.com ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@mg-d0.substack.com header.s=mailo header.b=HRSRG8PR; dkim=pass header.i=@mailgun.org header.s=mg header.b="H/y73NIk"; spf=pass (google.com: domain of bounce+91e48a.63af5d-tjphuhs=gmail.com@mg-d0.substack.com designates 161.38.202.253 as permitted sender) smtp.mailfrom="bounce+91e48a.63af5d-tjphuhs=gmail.com@mg-d0.substack.com"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=substack.com Return-Path: Received: from mg-202-253.substack.com (mg-202-253.substack.com. [161.38.202.253]) by mx.google.com with UTF8SMTPS id d2e1a72fcca58-83965646075si58600012b3a.39.2026.05.13.08.41.37 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 May 2026 08:41:37 -0700 (PDT) Received-SPF: pass (google.com: domain of bounce+91e48a.63af5d-tjphuhs=gmail.com@mg-d0.substack.com designates 161.38.202.253 as permitted sender) client-ip=161.38.202.253; Authentication-Results: mx.google.com; dkim=pass header.i=@mg-d0.substack.com header.s=mailo header.b=HRSRG8PR; dkim=pass header.i=@mailgun.org header.s=mg header.b="H/y73NIk"; spf=pass (google.com: domain of bounce+91e48a.63af5d-tjphuhs=gmail.com@mg-d0.substack.com designates 161.38.202.253 as permitted sender) smtp.mailfrom="bounce+91e48a.63af5d-tjphuhs=gmail.com@mg-d0.substack.com"; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=substack.com DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mg-d0.substack.com; q=dns/txt; s=mailo; t=1778686897; x=1778694097; h=List-Unsubscribe-Post: List-Unsubscribe: List-Post: List-Id: List-Archive: List-Owner: Reply-To: In-Reply-To: References: Sender: Sender: Date: Message-Id: To: To: From: From: Subject: Subject: Content-Type: Mime-Version: X-Feedback-Id; bh=LqUbMsWPbVOOcUkZuxru0itlB+kmYh6TpBY01tjBMws=; b=HRSRG8PRImZMHzgmukedmRqDryfDNKfJlycBkJZ/uZ3m/iltFAnl77hhEmP+HZ85LdGAPxnYiBTaBzavPxxGOv9PBZthNq71sC4dxahvdXGxu6hn+uKDBILHZymIm77LYEzVvxmoMWja89kPoBe8G3/OLc4xytJwgVPX1ejU1eA= DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mailgun.org; q=dns/txt; s=mg; t=1778686897; x=1778694097; h=List-Unsubscribe-Post: List-Unsubscribe: List-Post: List-Id: List-Archive: List-Owner: Reply-To: In-Reply-To: References: Sender: Sender: Date: Message-Id: To: To: From: From: Subject: Subject: Content-Type: Mime-Version: X-Feedback-Id; bh=LqUbMsWPbVOOcUkZuxru0itlB+kmYh6TpBY01tjBMws=; b=H/y73NIkZv53BM9bKBjL5NlUF9IadGrLWxBfKugq0LUcfMcf8/n9cCZjUQXN6jmHXHKMAvQjw4hErvENcwACy/hkKb8dyfQeGlGNiNJdu4Y1zRAxk8gqdgX1HssgsyJcfVcrCkg2iMOTlRJGsEueNvNO1PMib06kGQ+RacHOE4s= X-Mailgun-Sid: WyJkZWU4NyIsInRqcGh1aHNAZ21haWwuY29tIiwiNjNhZjVkIl0= X-Feedback-Id: postmaster@mg-d0.substack.com:post:6148c7d8a45790427cda307f:mailgun Received: by 3acb09075aed2b6d4898a615eca5c7f7f87624a0663d21f37eec85cfa2308b9a with HTTP id 6a049bb1efae4a7485db71be; Wed, 13 May 2026 15:41:37 GMT X-Mailgun-Sending-Ip: 161.38.202.253 Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="666e983acfd920f457cd93c2dc0675c6c14992e5b9e30eb00ffa296535ba" Subject: High Performance Rate Limiting at Databricks From: ByteByteGo To: tjphuhs@gmail.com X-Mailgun-Tag: post X-Mailgun-Track-Clicks: false Message-Id: <20260513153035.3.f13c7bbf59bd8f5c@mg-d0.substack.com> Date: Wed, 13 May 2026 15:30:35 +0000 Feedback-Id: post-196936594:cat-post:pub-817132:substack Sender: ByteByteGo References: In-Reply-To: Reply-To: ByteByteGo List-Owner: List-Url: List-Archive: List-Id: List-Post: List-Unsubscribe: List-Unsubscribe-Post: List-Unsubscribe=One-Click X-Mailgun-Variables: {"category": "post", "email_generated_at": "1778686897118", "is_freemail": "true", "post_audience": "everyone", "post_id": "196936594", "post_type": "newsletter", "pub_community_enabled": "true", "publication_id": "817132", "subdomain": "bytebytego", "user_id": "273622398"} --666e983acfd920f457cd93c2dc0675c6c14992e5b9e30eb00ffa296535ba Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable View this post on the web at https://blog.bytebytego.com/p/high-performance= -rate-limiting-at ScyllaDB Founders Share What Real-Time AI Requires from the Database (Spons= ored) [ https://substack.com/redirect/cc973a24-c008-4be2-a06b-86d5eca83aba?= j=3DeyJ1IjoiNGl3b2U2In0.sVDxRtmZ85v8kfdamY0krRXGMy3p768BWtuZifRB-Zs ] AI is pushing databases to their limits; learn what it takes to stay ahead AI workloads are exposing the limits of what most databases were designed t= o handle. Databases will need to process petabytes of data, millions of wri= tes per second, and data types like vectors =E2=80=93 all while delivering = consistent sub-millisecond P99 latency. Join ScyllaDB co-founders Dor Laor (CEO) and Avi Kivity (CTO) to explore wh= at real-time AI workloads actually require, and what it takes to stay ahead= =2E You will learn: How AI workloads are shifting in real-world applications The specific pressures these new patterns place on databases What architectural features help teams meet AI=E2=80=99s demands In early 2023, the Databricks rate limiter ran on a simple architecture. An= Envoy ingress gateway made calls to a Ratelimit Service, which in turn que= ried a single Redis instance. The setup handled the traffic it was designed= for, and the per-second nature of rate limiting meant the counts could sta= y transient without any durability guarantee. Then, the real-time model serving was launched. A single customer could now= generate orders of magnitude more traffic than the service was built for, = and three specific cracks appeared. Tail latency climbed sharply under load, worsened by two network hops and a= P99 of 10 to 20 milliseconds between services in one cloud provider. Adding machines and bolting on caches stopped helping past a certain point. The single Redis instance also represented a single point of failure that t= he team could no longer tolerate. The team redesigned the service, and the rebuild merits attention because t= he most interesting part is what they chose to give up. Strict accuracy is expensive at scale, and Databricks traded it for a faste= r critical path, a horizontally scalable counter, and a rate limiter that a= nswers as if the decision has already been made by the time the client chec= ks. In this article, we look at how Databricks implemented rate limiting at sca= le, how they shrank the critical path, and the accuracy tradeoff that shrin= king usually requires. Disclaimer: This post is based on publicly shared details from the Databric= ks Engineering Team. Please comment if you notice any inaccuracies. A Counting Problem Strip away the framing, and rate limiting reduces to a counting problem. Ea= ch request arrives, the system locates the right counter, compares it again= st a threshold, and either allows or rejects the request. The design questi= on is where that counter is stored and how quickly it works. In the old Databricks architecture, the counter was stored in Redis. See th= e diagram below: A request flowed through Envoy, hit the Ratelimit Service, and triggered a = call to Redis. That meant two network hops on the critical path of every re= quest. In a cloud provider where P99 network latency sat between 10 and 20 = milliseconds, those hops dominated the rate limit decision time. A check th= at should have cost microseconds was costing tens of milliseconds. See the diagram below: The team had already tried to work around this. Envoy can be configured wit= h consistent hashing so that requests with the same key land on the same Ra= telimit Service instance, which lets that instance keep a local count. The = approach helped, but it hit three walls. Non-Envoy services could not participate in the scheme, which fragmented th= e rate limit view. When the service cluster scaled up or restarted, the hash assignments churn= ed, which forced regular syncs back to Redis. Lastly, consistent hashing can be prone to hotspotting, where one very popu= lar key saturates a single machine while its neighbours sit idle. The only = way to push through those hotspots was to over-provision the entire cluster= =2E This is where scaling stops being additive. Adding machines stopped moving = the latency numbers, and adding more caching introduced more inconsistency.= The architecture itself was the ceiling, and the team had to change it. Moving the Count In-memory Rate limiting is transient. A per-second count exists only as long as that = second is current, and the moment it rolls over, the old value becomes irre= levant. That property opens a door. If a count only needs to live for a second, dur= able storage is more than the problem requires. The count can live in memor= y on the server that owns it, and losing that server during a restart costs= almost nothing. The challenge is that a single server cannot hold counts for every rate lim= it key across the fleet. The service needs a way to partition keys across s= ervers, and a way for any client to quickly find the server that owns a giv= en key. That is the problem Dicer solves at Databricks. For the purposes of this di= scussion, Dicer can be treated as a black box. Dicer is a routing layer tha= t lets a service keep state in memory while remaining horizontally scalable= and fault-tolerant. Clients ask Dicer which server owns a given key, and t= hat server confirms it is the authoritative owner before handling the reque= st. See the diagram below: With Dicer in place, the Ratelimit Service could move every counter in-memo= ry. The network hop to Redis disappeared. Server-side tail latency came dow= n sharply, and the team could scale horizontally by adding replicas to Dice= r=E2=80=99s assignment pool. The single point of failure went away as well,= because each replica became the authoritative store for its own slice of k= eys. Restarts and scale events redistributed ownership without any external= coordination, and the churn that had forced Redis syncs in the old consist= ent hashing setup stopped having an impact. This solved one problem and exposed another. The server side was now fast, but the client side still made a synchronous = call across the network for every single request. A rate limit check that h= ad been waiting on Redis was now waiting on the Rate Limit Service. The P99= came down, but the shape of the problem remained. The critical path still = had a round trip on it. Optimistic Rate Limiting This is where the team made its most consequential move. Millions of client requests per second were still translating into millions= of synchronous calls to the Ratelimit Service. Even with the server answer= ing in memory, that represented significant network traffic, significant se= rver capacity, and significant client-side waiting. The team asked a harder= question. Does every request truly need to wait for a rate limit decision before proc= eeding? They considered three alternatives: The first was prefetching tokens on the client, where the client pulls a bl= ock of capacity and answers rate limit checks locally. The second was batching requests on the client and waiting for a response b= efore releasing them. The third was sampling, where only a fraction of requests get checked. Each had problems. Prefetching carries messy edge cases during startup, exp= iry, and token exhaustion. Batching adds delay and memory pressure. Samplin= g works for high-QPS limits but falls apart when the limit itself is small. What the team built is called batch-reporting, and it rests on two ideas: The first is that clients make no remote calls on the rate limit path. The second is optimistic rate limiting, where the default is to allow the r= equest and reject only when the client already has a reason to reject from = an earlier report. The client counts how many requests it let through and how many it rejected= , grouped by rate limit key. Every 100 milliseconds or so, a background thr= ead packages those counts and reports them to the Ratelimit Service. The se= rver responds with instructions telling the client which keys should be rej= ected, until which timestamp, and at what rejection rate. The diagram below shows the three architectures side by side: The impact was substantial: Tail latency on rate limit calls fell by roughly a factor of ten, because t= he calls were effectively free for the client. Spiky inbound traffic turned into constant outbound reports because the rep= orting cadence is fixed regardless of how bursty the underlying traffic bec= omes. Server-side load became predictable for the first time. The inversion merits emphasis. The rate limiter used to be asked before eac= h decision. Now it is told after. That inversion sits at the core of the re= design, and the tradeoff is explicit. Databricks accepts that some requests= over the limit will slip through between reports, and their backends are b= uilt to tolerate that overshoot. Bounding The Overshoot Batch-reporting introduced a problem of its own. Between the moment a client starts exceeding a limit and the moment the ser= ver tells it to reject, traffic can leak through. A hundred milliseconds of= overshoot at high QPS amounts to a lot of requests. The team wanted guaran= tees that kept overshoot within roughly 5 percent of the policy, and reachi= ng that target required three-layered fixes. The first was a rejection rate returned by the server. The idea is to use t= he past to predict the near future. If the last second=E2=80=99s traffic ex= ceeded the policy by some amount, the formula rejectionRate equals (estimat= edQps minus rateLimitPolicy) divided by estimatedQps tells the client what = fraction of upcoming requests to drop. This assumes that the next second=E2= =80=99s traffic resembles the last second=E2=80=99s, which often holds true= to help. The second was a client-side local rate limiter as a defense in depth. When= traffic spikes so obviously that the batch cycle has no chance of catching= up, the client starts rejecting locally based on its own counts. This catc= hes the extreme cases immediately rather than waiting for a round trip. The third was the algorithm itself. Once autosharding lets the service hold= counts in memory, the token bucket becomes feasible. The token bucket has = a useful property that a fixed window lacks. Fixed window resets to zero at= the end of every interval, so a customer can blast traffic right at a wind= ow boundary and technically stay within the policy while sending double the= intended rate during the crossover. The token bucket continuously fills an= d drains, and it can go negative. When a customer sends too many requests, = the bucket remembers and stays empty until the refill catches up. The reset= problem disappears. Token bucket also approximates a sliding window when c= onfigured without extra burst capacity, which produces a stricter shape tha= n a fixed window for most limits. This is where the algorithm choice was gated by the storage choice. Token b= ucket needs compare-and-set style logic on every increment, which was slow = in Redis. In-memory, the same operation is close to free. Once the token bu= cket was viable, the earlier rejection rate mechanism became unnecessary, a= nd the team eventually converted every rate limit in the system to a token = bucket. Three Coupled Decisions The Databricks story resolves into three decisions that depend on each othe= r: The first is the algorithm, which determines how the counter behaves at the= boundaries of time intervals. Fixed window, sliding window, and token buck= et each produce different behaviour in that regard. The second is where the state lives, whether in a shared external store lik= e Redis, in an in-memory counter on a single server, or in an in-memory cou= nter sharded across a cluster of servers through some routing layer. The third is the sync model, which can either require every request to wait= for a synchronous decision or allow clients to make local decisions and re= concile through asynchronous reports. The old Databricks architecture sat at one corner of that space, combining = a fixed window, shared Redis, and synchronous per-request checks. The new a= rchitecture sits at a different corner, combining token bucket, sharded in-= memory storage, and asynchronous batch reports. The dependency chain is worth understanding. Token bucket needs cheap compa= re-and-set semantics, which rules out Redis at the QPS Databricks sees, whi= ch forces in-memory state. In-memory state across many counters forces shar= ding, because one server cannot hold everything. Sharding with authoritativ= e per-key ownership is what enables batch-reporting, because each shard can= act as the source of truth for its keys without coordinating with peers. These constraints explain the order of the rollout. Sharded in-memory came = first, batch-reporting followed on top of it, and token bucket replaced the= algorithm once the state architecture could support it. Conclusion The end result is a rate limiter that is faster, more resilient, and more s= calable than what came before, at the cost of strict accuracy. That tradeof= f is the foundation of the design. A system that has to enforce limits exactly, because each request over the = limit costs real money or violates a contract, would have to pick a differe= nt architecture. Databricks could afford this one because their backends to= lerate roughly 5 percent overshoot. A few smaller details from the rollout are worth noting. The team built a l= ocalhost sidecar next to the Envoy ingress to host the batch-reporting logi= c, because Envoy is third-party code they could not change directly. Before= in-memory counting was ready, a Lua script on Redis batched writes togethe= r to keep batch-reporting latency manageable during the migration. The rebuild reframes what rate limiting is as a system problem. The algorithm tends to get the attention, but the storage and sync model de= termine whether the algorithm can run at scale. Distributed counting is a s= ingle design problem with three coupled aspects rather than three independe= nt ones. References: High Performance Ratelimiting at Databricks [ https://substack.com/redirect= /b90befee-bbce-4d6e-85b9-5921c48697b7?j=3DeyJ1IjoiNGl3b2U2In0.sVDxRtmZ85v8k= fdamY0krRXGMy3p768BWtuZifRB-Zs ] Unsubscribe https://substack.com/redirect/2/eyJlIjoiaHR0cHM6Ly9ibG9nLmJ5dGV= ieXRlZ28uY29tL2FjdGlvbi9kaXNhYmxlX2VtYWlsP3Rva2VuPWV5SjFjMlZ5WDJsa0lqb3lOek= 0yTWpJek9UZ3NJbkJ2YzNSZmFXUWlPakU1Tmprek5qVTVOQ3dpYVdGMElqb3hOemM0TmpnMk9Ea= zNMQ0psZUhBaU9qRTRNVEF5TWpJNE9UY3NJbWx6Y3lJNkluQjFZaTA0TVRjeE16SWlMQ0p6ZFdJ= aU9pSmthWE5oWW14bFgyVnRZV2xzSW4wLmQySDBOenh3b1JsU3pLX3dzZHBCVFkzS3dNNkdYc0t= ZdEU2TFBtd28tS3MiLCJwIjoxOTY5MzY1OTQsInMiOjgxNzEzMiwiZiI6dHJ1ZSwidSI6MjczNj= IyMzk4LCJpYXQiOjE3Nzg2ODY4OTcsImV4cCI6MjA5NDI2Mjg5NywiaXNzIjoicHViLTAiLCJzd= WIiOiJsaW5rLXJlZGlyZWN0In0.fOLHTBQexV2x2a83iBNke3lzIJMcoNb4tZd33PcWWfE? --666e983acfd920f457cd93c2dc0675c6c14992e5b9e30eb00ffa296535ba Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable = High Performance Rate Limiting at Databricks=3D""
In this article, we look at how Databricks imp= lemented rate limiting at scale, how they shrank the critical path, and the= accuracy tradeoff that shrinking usually requires.
͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­͏     ­͏   ̳= 9; ­͏     ­͏     ­͏ &nb= sp;   ­͏     ­͏     ­&= #847;     ­͏     ­͏    = ; ­͏     ­͏     ­͏ &nbs= p;   ­͏     ­͏     ­&#= 847;     ­͏     ­͏    = ­͏     ­͏     ­͏  = ;   ­͏     ­͏     ­= 47;     ­͏     ­͏     = ­͏     ­͏     ­͏  =   ­͏     ­͏     ­T= 7;     ­͏     ­͏     &= #173;͏     ­͏     ­͏   =   ­͏     ­͏     ­͏= ;     ­͏     ­͏     &#= 173;͏     ­͏     ­͏   &= #8199; ­͏     ­͏     ­͏=     ­͏     ­͏     = 73;͏     ­͏     ­͏   &#= 8199; ­͏     ­͏     ­͏ =     ­͏     ­͏     = 3;͏     ­͏     ­͏   = 199; ­͏     ­͏     ­͏ &= nbsp;   ­͏     ­͏     ­= ;͏     ­͏     ­͏   Q= 99; ­͏     ­͏     ­͏ &n= bsp;   ­͏     ­͏     ­= ͏     ­
Forwar= ded this email? Subscribe here for more
&n= bsp;
3D""
= 3D""
3D""
READ IN APP3D""
 

ScyllaDB Founders = Share What Real-Time AI Requires from the Database (Sponsored)

AI is pushing databases to their limits; learn wha= t it takes to stay ahead

=
<= /td>

AI workloads are expos= ing the limits of what most databases were designed to handle. Databases wi= ll need to process petabytes of data, millions of writes per second, and da= ta types like vectors – all while delivering consistent sub-milliseco= nd P99 latency.

Join ScyllaDB co-founders Dor Laor (CEO) an= d Avi Kivity (CTO) to explore what real-time AI workloads actually require,= and what it takes to stay ahead.

You will learn:

  • How AI workloads are shifting in real-world applications

    • <= li style=3D"margin: 8px 0 0 32px;mso-special-format: bullet;">

    The specific pressures the= se new patterns place on databases

  • What architectural features help teams meet AI’s de= mands

Register for Free

= In early 2023, the Databricks rate limiter ran on a simple architecture. An= Envoy ingress gateway made calls to a Ratelimit Service, which in turn que= ried a single Redis instance. The setup handled the traffic it was designed= for, and the per-second nature of rate limiting meant the counts could sta= y transient without any durability guarantee.

Then, the real-time model serving was launched. A single customer cou= ld now generate orders of magnitude more traffic than the service was built= for, and three specific cracks appeared.

  • Tail latency climbed sharply under load, worsened by two network hops a= nd a P99 of 10 to 20 milliseconds between services in one cloud provider.

  • Adding machines and bolting on caches stopped helping past a certain point= =2E

  • The single Redis instance also represented a single point of failure th= at the team could no longer tolerate.

The team redesigned the service, and the rebuild merits attention b= ecause the most interesting part is what they chose to give up.

Strict accuracy is expensive at scale, and Databric= ks traded it for a faster critical path, a horizontally scalable counter, a= nd a rate limiter that answers as if the decision has already been made by = the time the client checks.

In this arti= cle, we look at how Databricks implemented rate limiting at scale, how they= shrank the critical path, and the accuracy tradeoff that shrinking usually= requires.

Disclaimer: This post is = based on publicly shared details from the Databricks Engineering Team. Plea= se comment if you notice any inaccuracies.

A Counting Problem

= Strip away the framing, and rate limiting reduces to a counting problem. Ea= ch request arrives, the system locates the right counter, compares it again= st a threshold, and either allows or rejects the request. The design questi= on is where that counter is stored and how quickly it works.

In the old Databricks architecture, the counter was st= ored in Redis. See the diagram below:

= <= td style=3D"text-align: center;">
3D""
<= p style=3D"margin: 0 0 20px 0;color: rgb(54,55,55);line-height: 26px;font-s= ize: 16px;text-align: justify">A request flowed through Envoy, hit the Rate= limit Service, and triggered a call to Redis. That meant two network hops o= n the critical path of every request. In a cloud provider where P99 network= latency sat between 10 and 20 milliseconds, those hops dominated the rate = limit decision time. A check that should have cost microseconds was costing= tens of milliseconds.

See the diagram b= elow:

=

Th= e team had already tried to work around this. Envoy can be configured with = consistent hashing so that requests with the same key land on the same Rate= limit Service instance, which lets that instance keep a local count. The ap= proach helped, but it hit three walls.

  • Non-Envoy services could not participate in the scheme, which fragmented t= he rate limit view.

  • When the service cluster scaled up or restarted, the = hash assignments churned, which forced regular syncs back to Redis.

  • Lastl= y, consistent hashing can be prone to hotspotting, where one very popular k= ey saturates a single machine while its neighbours sit idle. The only way t= o push through those hotspots was to over-provision the entire cluster.

    =

This is where scaling stops being= additive. Adding machines stopped moving the latency numbers, and adding m= ore caching introduced more inconsistency. The architecture itself was the = ceiling, and the team had to change it.

Moving the Count In-memory

= Rate limiting is transient. A per-second count exists only as long as that = second is current, and the moment it rolls over, the old value becomes irre= levant.

That property opens a door. If a= count only needs to live for a second, durable storage is more than the pr= oblem requires. The count can live in memory on the server that owns it, an= d losing that server during a restart costs almost nothing.

The challenge is that a single server cannot hold count= s for every rate limit key across the fleet. The service needs a way to par= tition keys across servers, and a way for any client to quickly find the se= rver that owns a given key.

That is the = problem Dicer solves at Databricks. For the purposes of this discussion, Di= cer can be treated as a black box. Dicer is a routing layer that lets a ser= vice keep state in memory while remaining horizontally scalable and fault-t= olerant. Clients ask Dicer which server owns a given key, and that server c= onfirms it is the authoritative owner before handling the request.

See the diagram below:

3D""
<= /figure>

With Dicer in place, the Rate= limit Service could move every counter in-memory. The network hop to Redis = disappeared. Server-side tail latency came down sharply, and the team could= scale horizontally by adding replicas to Dicer’s assignment pool. Th= e single point of failure went away as well, because each replica became th= e authoritative store for its own slice of keys. Restarts and scale events = redistributed ownership without any external coordination, and the churn th= at had forced Redis syncs in the old consistent hashing setup stopped havin= g an impact.

This solved one problem and= exposed another.

The server side was no= w fast, but the client side still made a synchronous call across the networ= k for every single request. A rate limit check that had been waiting on Red= is was now waiting on the Rate Limit Service. The P99 came down, but the sh= ape of the problem remained. The critical path still had a round trip on it= =2E

Optimistic Rate Limiting

This is where the team made its most con= sequential move.

Millions of client requ= ests per second were still translating into millions of synchronous calls t= o the Ratelimit Service. Even with the server answering in memory, that rep= resented significant network traffic, significant server capacity, and sign= ificant client-side waiting. The team asked a harder question.

Does every request truly need to wait for a rate li= mit decision before proceeding?

They con= sidered three alternatives:

  • The first = was prefetching tokens on the client, where the client pulls a block of cap= acity and answers rate limit checks locally.

  • The second was batching requ= ests on the client and waiting for a response before releasing them.

  • Th= e third was sampling, where only a fraction of requests get checked.

Each had problems. Prefetching carri= es messy edge cases during startup, expiry, and token exhaustion. Batching = adds delay and memory pressure. Sampling works for high-QPS limits but fall= s apart when the limit itself is small.

= What the team built is called batch-reporting, and it rests on two ideas:

  • The first is that clients make no remot= e calls on the rate limit path.

  • The second is optimistic rate limiting, w= here the default is to allow the request and reject only when the client al= ready has a reason to reject from an earlier report.

= 3D""=

The client counts ho= w many requests it let through and how many it rejected, grouped by rate li= mit key. Every 100 milliseconds or so, a background thread packages those c= ounts and reports them to the Ratelimit Service. The server responds with i= nstructions telling the client which keys should be rejected, until which t= imestamp, and at what rejection rate.

Th= e diagram below shows the three architectures side by side:

<= td class=3D"content" align=3D"left" width=3D"1456" style=3D"text-align: cen= ter;">3D""=

The impact was subst= antial:

  • Tail latency on rate limit cal= ls fell by roughly a factor of ten, because the calls were effectively free= for the client.

  • Spiky inbound traffic turned into constant outbound repo= rts because the reporting cadence is fixed regardless of how bursty the und= erlying traffic becomes.

  • Server-side load became predictable for the firs= t time.

The inversion merits e= mphasis. The rate limiter used to be asked before each decision. Now it is = told after. That inversion sits at the core of the redesign, and the tradeo= ff is explicit. Databricks accepts that some requests over the limit will s= lip through between reports, and their backends are built to tolerate that = overshoot.

Bounding The Over= shoot

Batch-reporting introduced a prob= lem of its own.

Between the moment a cli= ent starts exceeding a limit and the moment the server tells it to reject, = traffic can leak through. A hundred milliseconds of overshoot at high QPS a= mounts to a lot of requests. The team wanted guarantees that kept overshoot= within roughly 5 percent of the policy, and reaching that target required = three-layered fixes.

The first was a rej= ection rate returned by the server. The idea is to use the past to predict = the near future. If the last second’s traffic exceeded the policy by = some amount, the formula rejectionRate equals (estimatedQps minus rateLimit= Policy) divided by estimatedQps tells the client what fraction of upcoming = requests to drop. This assumes that the next second’s traffic resembl= es the last second’s, which often holds true to help.

The second was a client-side local rate limiter as a de= fense in depth. When traffic spikes so obviously that the batch cycle has n= o chance of catching up, the client starts rejecting locally based on its o= wn counts. This catches the extreme cases immediately rather than waiting f= or a round trip.

3D""

The third was the algorithm itself. Once autosharding lets the= service hold counts in memory, the token bucket becomes feasible. The toke= n bucket has a useful property that a fixed window lacks. Fixed window rese= ts to zero at the end of every interval, so a customer can blast traffic ri= ght at a window boundary and technically stay within the policy while sendi= ng double the intended rate during the crossover. The token bucket continuo= usly fills and drains, and it can go negative. When a customer sends too ma= ny requests, the bucket remembers and stays empty until the refill catches = up. The reset problem disappears. Token bucket also approximates a sliding = window when configured without extra burst capacity, which produces a stric= ter shape than a fixed window for most limits.

This is where the algorithm choice was gated by the storage choice. = Token bucket needs compare-and-set style logic on every increment, which wa= s slow in Redis. In-memory, the same operation is close to free. Once the t= oken bucket was viable, the earlier rejection rate mechanism became unneces= sary, and the team eventually converted every rate limit in the system to a= token bucket.

Three Coupled= Decisions

The Databricks story resolve= s into three decisions that depend on each other:

  • The first is the algorithm, which determines how the counter be= haves at the boundaries of time intervals. Fixed window, sliding window, an= d token bucket each produce different behaviour in that regard.

    • The secon= d is where the state lives, whether in a shared external store like Redis, = in an in-memory counter on a single server, or in an in-memory counter shar= ded across a cluster of servers through some routing layer.

  • The third is = the sync model, which can either require every request to wait for a synchr= onous decision or allow clients to make local decisions and reconcile throu= gh asynchronous reports.

The o= ld Databricks architecture sat at one corner of that space, combining a fix= ed window, shared Redis, and synchronous per-request checks. The new archit= ecture sits at a different corner, combining token bucket, sharded in-memor= y storage, and asynchronous batch reports.

The dependency chain is worth understanding. Token bucket needs cheap co= mpare-and-set semantics, which rules out Redis at the QPS Databricks sees, = which forces in-memory state. In-memory state across many counters forces s= harding, because one server cannot hold everything. Sharding with authorita= tive per-key ownership is what enables batch-reporting, because each shard = can act as the source of truth for its keys without coordinating with peers= =2E

These constraints explain the order of= the rollout. Sharded in-memory came first, batch-reporting followed on top= of it, and token bucket replaced the algorithm once the state architecture= could support it.

Conclusio= n

The end result is a rate limiter that= is faster, more resilient, and more scalable than what came before, at the= cost of strict accuracy. That tradeoff is the foundation of the design.

A system that has to enforce limits exactl= y, because each request over the limit costs real money or violates a contr= act, would have to pick a different architecture. Databricks could afford t= his one because their backends tolerate roughly 5 percent overshoot.

A few smaller details from the rollout are wor= th noting. The team built a localhost sidecar next to the Envoy ingress to = host the batch-reporting logic, because Envoy is third-party code they coul= d not change directly. Before in-memory counting was ready, a Lua script on= Redis batched writes together to keep batch-reporting latency manageable d= uring the migration.

The rebuild reframe= s what rate limiting is as a system problem.

The algorithm tends to get the attention, but the storage and sync mod= el determine whether the algorithm can run at scale. Distributed counting i= s a single design problem with three coupled aspects rather than three inde= pendent ones.

References:

 
<= /tbody>
3D""Share
 
 
3D""Like
3D""Comment
=
3D""= Restack=
 

© 2026 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104 =
Unsubscribe

<= p class=3D"footerSection-EHR0jG small powered-by-substack" style=3D"padding= : 0 24px;font-size: 12px;line-height: 20px;margin: 0;color: rgb(119,119,119= );font-family: system-ui,-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto= ,Helvetica,Arial,sans-serif,'Apple Color Emoji','Segoe UI Emoji','Segoe UI = Symbol';padding-bottom: 0;margin-top: 0;">3D"Start

3D""3D"" --666e983acfd920f457cd93c2dc0675c6c14992e5b9e30eb00ffa296535ba--