One Distributed SQL Write Path’s Latency Tax Surprised a Team’s Monthly Storage Bill

When a mid-size SaaS company moved its write-heavy workload to a nine-node CockroachDB cluster spread across three AWS regions, the team expected higher costs. What they did not expect was a storage bill that grew 40% month over month with no corresponding increase in user traffic. The culprit was not data volume or a misconfigured retention policy. It was something far subtler: the latency tax buried in every distributed SQL write path.

A Monthly Bill That Wouldn’t Stop Climbing

The company ran a real-time analytics platform that ingested event streams from hundreds of sources. Each event triggered a write to CockroachDB. The cluster had nine nodes — three in us-east-1, three in us-west-2, and three in eu-west-1. The team chose CockroachDB for its strong consistency and automatic rebalancing. They did not choose it for its cost profile, but that profile soon dominated their monthly infrastructure reviews.

In month one, the bill was roughly $12,000. Month two hit $17,000. By month three, it exceeded $24,000. The finance team flagged it. Engineering started digging. What they found was not a leaky bucket of storage but a slow bleed caused by network latency on every write. Each write forced a round trip across the Atlantic before it could commit. The cluster was spending more time waiting for network packets than processing data.

The team’s first instinct was to blame the cloud provider. But AWS’s network metrics showed no anomalies. The second instinct was to blame CockroachDB itself. But the database was behaving exactly as designed — the design just happened to be expensive for their workload. The problem was not a bug. It was a mismatch between the default configuration and the physical topology of the deployment.

As one engineer put it: “We treated the cluster as a single logical database, but every write was paying a cross-region tax. The bill was just the receipt for that tax.”

The Write Path Anatomy That Everyone Skips

Distributed SQL databases like CockroachDB, YugabyteDB, and Google Spanner use a consensus protocol — typically Raft or Paxos — to ensure that every write is durably replicated before acknowledging success. In CockroachDB, data is split into ranges, each of which runs its own Raft group. A write must be proposed to the leaseholder of the range, which then replicates the write to a majority of the voting replicas in that group.

In a single-region deployment, that round trip takes roughly 1 to 5 milliseconds. The leaseholder and voting replicas sit in the same datacenter, connected by low-latency links. The write path is short. But in a multi-region setup, the story changes. If the leaseholder is in us-east-1 and two voting replicas are in us-west-2 and eu-west-1, the write must replicate to at least two out of three replicas — meaning the leaseholder waits for acknowledgments from at least one remote region.

The latency tax is the difference between the local RTT and the remote RTT. For us-east-1 to eu-west-1, that is roughly 75 to 90 milliseconds round trip. For us-east-1 to us-west-2, it is roughly 60 to 70 milliseconds. Every write pays at least one of those round trips. Over thousands of writes per second, the tax compounds into seconds of cumulative latency — and seconds translate directly into compute and network resource consumption.

Most teams do not think about this until the bill arrives. The database abstracts away the physical layout, but the physics of light in fiber does not abstract. Every cross-region write incurs the speed-of-light penalty. And because consensus requires a majority, the slowest replica in the quorum sets the pace. The write path is only as fast as the farthest necessary hop.

Where the Latency Tax Turns Into a Dollar Sign

Latency does not directly appear on a cloud bill. But it drives two cost factors: compute utilization and network egress. When a write takes 80 milliseconds instead of 5, the database holds open connections longer, consumes CPU cycles while waiting, and uses more memory for pending Raft entries. These costs add up.

In CockroachDB, each range’s leaseholder handles all writes for that range. If the leaseholder is far from the client, every write incurs a network round trip before the database even begins processing. The team’s workload averaged roughly 2,000 writes per second. With a 60-millisecond round trip, that is 120 seconds of cumulative latency per second — meaning the cluster was effectively running at 120% of its theoretical capacity just due to waiting.

The cloud provider charges for sustained CPU usage above a threshold. The team saw their instance types upgraded from m5.large to m5.xlarge within two months, not because they needed more storage but because the CPU was pinned at 70% due to network waits. The cost of those larger instances was roughly double. And network egress between regions added another 5–10% to the bill.

Meanwhile, reads were cheap. CockroachDB supports follower reads — serving read requests from any replica without going through the leaseholder. The team’s read-to-write ratio was about 3:1, but reads were not the problem. Writes were the anchor dragging down both performance and cost. The team had inadvertently optimized for reads while ignoring the write path entirely.

The Team’s Mistake: Default Configuration in a Multi-Region Setup

The team had launched their cluster with CockroachDB’s default configuration. They set the replication factor to 3 and let the database automatically place replicas across the three regions. This meant each range had three voting replicas — one in each region. The leaseholder could be in any region. In practice, it often ended up in us-east-1 because that region received the most client traffic.

But the clients were distributed. A client in eu-west-1 sending a write to a range whose leaseholder was in us-east-1 would pay a 75-millisecond round trip before the write even reached the leaseholder. Then the leaseholder had to replicate to a majority — which required at least one additional replica, often in us-west-2 or eu-west-1. That added another 60-millisecond hop. Total write latency: around 135 milliseconds. For a workload that expected under 10 milliseconds.

The team did not realize that CockroachDB’s default configuration assumes a single-region or close-region deployment. The database’s automatic rebalancing tries to spread replicas evenly, which in a multi-region setup guarantees that every write crosses at least one region boundary. The default is safe — it provides maximum fault tolerance — but it is not economical for write-heavy workloads.

They also did not use non-voting replicas. Non-voting replicas hold a copy of the data but do not participate in the Raft quorum. They are useful for serving reads locally without slowing down writes. But the team had not read that part of the documentation. They assumed all replicas were equal. They were not.

The result was a system that spent most of its time waiting on network I/O. The CPU was idle during those waits, but the clock was ticking. And the cloud provider was charging for every tick.

Fixing It: Non-Voting Replicas and Region-Specific Leaseholders

Once the team identified the root cause, the fix was conceptually simple: reduce the number of cross-region hops required for each write. They did two things. First, they reconfigured the replication strategy so that each range had one voting replica in the primary region (us-east-1) and two non-voting replicas in the other regions. The quorum now required only one replica — the local leaseholder — so writes committed without waiting for a remote acknowledgment.

Second, they pinned the leaseholder to the region where the majority of writes originated. Using CockroachDB’s ALTER TABLE ... CONFIGURE ZONE with leaseholder_preferences, they ensured that the leaseholder for write-heavy ranges always lived in us-east-1. This eliminated the initial cross-region hop for clients in that region. Clients in other regions still paid a latency penalty, but their write volume was lower.

The effect was dramatic. Write latency dropped from roughly 60 milliseconds to about 10 milliseconds for the majority of writes. CPU utilization on the cluster nodes fell from 70% to around 30%. The team was able to downgrade their instance sizes back to m5.large, cutting compute costs by nearly half. The monthly bill stabilized at around $13,000 — slightly above the original $12,000 but no longer growing exponentially.

The team also enabled follower reads for their read-heavy queries, further reducing load on the leaseholder. But the main savings came from the write path fix. As one engineer noted, “We were paying for latency we didn’t need. The database could handle our throughput; it just needed to stop waiting for packets from London.”

The whole reconfiguration took about two days of testing in a staging environment and a rolling update to production. No code changes were needed. The fix was entirely operational.

The Real Lesson: Treat Distributed SQL Like a Cost Model

The team’s experience is not unique. Distributed SQL databases are powerful, but they come with a cost model that is easy to ignore until the bill arrives. The write path is the most expensive part of that model because it involves consensus, and consensus involves network round trips. Every cross-region hop adds latency, and latency burns money.

The lesson is to map your write path to your physical network topology before you launch. Identify where your clients are, where the leaseholders will be, and how many cross-region hops each write will require. CockroachDB’s DB Console provides per-range latency metrics that can help. Run a realistic workload in staging and measure the latency distribution. If the tail is long, your bill will be too.

Also consider whether you need strong consistency for every write. If your application can tolerate eventual consistency for some operations, a simpler caching layer or a NoSQL database might be cheaper. Distributed SQL is a tool, not a default. Use it where the tradeoff is worth it.

For teams already running a distributed SQL cluster, the same analysis applies. Look at your write latency by range. If you see high variance, check whether leaseholders are far from clients. If you see consistent high latency, consider non-voting replicas or leaseholder pinning. The fix is often cheaper than the status quo.

As one engineer on the team put it: “We thought we needed a distributed database for scale. What we actually needed was a distributed database for availability. We could have achieved our throughput with a single node and a read replica. The multi-region setup was for disaster recovery, not for performance.” That distinction is critical.

When to Avoid Distributed SQL Altogether

The SaaS team’s story raises a broader question: when should you not use distributed SQL? The answer depends on your workload. If your write throughput is under 1,000 operations per second, a single-node PostgreSQL or MySQL instance with a read replica will almost certainly be cheaper and faster. The overhead of consensus is not justified at that scale.

If your workload is write-heavy and latency-sensitive, distributed SQL may never be the right choice. The physics of consensus imposes a floor on write latency that cannot be eliminated. Even with leaseholder pinning and non-voting replicas, a write to a local leaseholder still takes 5–10 milliseconds — compared to sub-millisecond for a single-node database. For applications that need 99th percentile write latency under 5 milliseconds, a monolithic database with application-level sharding is often a better fit.

Geo-distribution makes sense only if reads dominate writes, or if the business absolutely requires strong consistency across regions for every write. For the SaaS team, only a small fraction of their data needed that guarantee. Most of their writes could have been eventually consistent. They chose CockroachDB for its operational simplicity, but that simplicity came with a cost premium.

YugabyteDB and Google Spanner have similar tradeoffs. YugabyteDB uses the same Raft-based write path, and Spanner uses Paxos with TrueTime. Both require careful configuration for multi-region deployments. Spanner’s regional instances avoid cross-region writes but sacrifice availability during a regional outage. There is no free lunch.

Ultimately, the decision to use distributed SQL should be driven by a cost model, not by hype. Map your write path. Measure your latency. Calculate the tax. If the tax exceeds the value of the guarantees, reconsider. The SaaS team’s bill was a wake-up call. It could be yours too.