$
~/posts/2025-04-04_understanding-and-scaling-raft.md
$

cat 2025-04-04_understanding-and-scaling-raft.md

📅

If you have worked with distributed systems, you must have come across this famous quote

In a distributed system, the only thing two nodes can agree on is that they can't agree on anything.

This quote has stuck with me because it perfectly captures the complexity of building distributed systems. The challenge of getting multiple machines to agree on a single value is so fundamental that it has spawned entire fields of research. Today, we'll look at one elegant solution to this problem - the Raft consensus algorithm.

Let's break this down with some real examples and then regroup to understand the implications.

Example 1 - The Three Generals

Imagine three generals commanding different divisions of an army. They need to decide whether to attack or retreat, but they can only communicate through messengers who might get captured.

This is exactly the problem Raft solves, but instead of generals, we have servers, and instead of messengers, we have network packets. The genius of Raft lies in how it simplifies this complex problem.

In Raft, a leader is elected and becomes the source of truth. Other nodes follow.

The Core Ideas

Leader Election

Think of this like a democratic election, but with fixed rules:

  • Each server starts as a Follower
  • If no leader is heard from, a Follower becomes a Candidate
  • The Candidate requests votes from others
  • If it gets majority votes, it becomes the Leader
  • If two candidates split the vote, a new election starts

Log Replication

This is where Raft really shines. Instead of having every node try to agree on everything:

  • The leader receives all client requests
  • It adds them to its log
  • It replicates this log to followers
  • Once majority confirm, the entry is committed

It's like a game of "Simon Says" - the leader (Simon) gives commands, and others follow.

Scaling Challenges

The Cost of Consensus

There's a reason why we don't use Raft for everything. Each write operation needs:

  • 1 round-trip to leader
  • 1 round-trip to followers
  • 1 round-trip back to client

This means in a 5-node cluster, a single write operation involves at least 3 network round-trips.

The Multi-Raft Pattern

This is where it gets interesting. Instead of one large Raft group:

  • Split data into ranges/shards
  • Each range has its own Raft group
  • Different leaders for different ranges

This is how CockroachDB and TiDB handle scale - they run thousands of Raft groups in parallel.

CockroachDB's Multi-Raft Optimizations

CockroachDB takes the Multi-Raft pattern further with several clever optimizations:

Colocated Raft Groups

Multiple Raft groups on the same node share resources intelligently

Instead of treating each Raft group as completely independent:

  • Shared thread pools for processing
  • Batched disk writes across groups
  • Unified network connections between nodes
  • Shared memory pools for better resource utilization

Message Batching

CockroachDB batches messages across Raft groups:

  • Multiple heartbeats combined into single network packets
  • Log entries from different groups bundled together
  • Responses aggregated for efficient network usage
  • Smart throttling to prevent overwhelming nodes

Range Leases

To reduce Raft overhead further, CockroachDB introduces Range Leases:

  • Long-term (seconds) read delegation to a single node
  • Reads don't need full Raft consensus
  • Lease transfers are still Raft-coordinated
  • Significantly reduces read latency

Dynamic Range Split and Merge

The system automatically manages Raft groups:

  • Hot ranges split automatically
  • Cold ranges merge to reduce overhead
  • Load-based splitting for better resource distribution
  • Geography-aware range distribution

This is particularly powerful because:

  • Reduces operational complexity
  • Automatically handles hot spots
  • Maintains optimal range sizes
  • Balances between too many vs too few Raft groups

The Dark Side of Multi-Raft

While Multi-Raft solves scaling problems, it introduces its own set of challenges:

Complexity Explosion

With great power comes great operational complexity

Running thousands of Raft groups means:

  • More state to track and debug
  • Complex failure scenarios
  • Harder to reason about system behavior
  • Increased monitoring overhead

Resource Management Nightmares

Each Raft group needs:

  • Memory for in-flight operations
  • Disk space for logs
  • Network bandwidth for replication
  • CPU for leader election and log processing

When you have thousands of groups, resource spikes can cascade:

  • One slow disk affects multiple groups
  • Network congestion impacts all colocated groups
  • Memory pressure causes widespread slowdowns

Cross-Range Transactions

When data spans multiple ranges:

  • Need atomic commits across Raft groups
  • Complex coordination protocols required
  • Higher latency for distributed transactions
  • Increased chance of conflicts and retries

Testing Challenges

Testing becomes exponentially harder:

  • Need to simulate thousands of groups
  • Complex failure scenarios multiply
  • Race conditions become more subtle
  • Performance testing requires real scale

This is why you'll often hear:

Don't use Multi-Raft unless you absolutely need the scale

Let's regroup

What makes Raft special isn't just its technical merits, but how it makes distributed consensus understandable. It takes the complex problem of distributed agreement and breaks it down into manageable pieces:

  • Leader election
  • Log replication
  • Safety guarantees

What to watch out for?

Network Partitions

The famous quote "Networks are reliable" was listed as the first fallacy of distributed computing for a reason. Network partitions can cause:

  • Split brain scenarios
  • Temporary unavailability
  • Increased latency

State Machine Size

Your Raft log can't grow forever. You need to think about:

  • Compaction strategies
  • Snapshot shipping
  • Recovery mechanisms

Membership Changes

Adding or removing nodes from a Raft cluster is tricky:

  • Can't change multiple nodes at once
  • Need to maintain quorum during transitions
  • Must handle failures during membership changes

The beauty of Raft lies in its simplicity. While other consensus algorithms might be more efficient in specific scenarios, Raft's understandability makes it the go-to choice for many distributed systems.

Remember:

Consensus is expensive. Use it only when you absolutely need it.

Take the time to understand when you need strong consistency versus when eventual consistency is good enough. Your system's scalability might depend on it.

Hope you liked reading the article.

Please reach out to me here for more ideas or improvements.

cd ..