Review & Cheat Sheet

[!IMPORTANT] In this lesson, you will master:

  1. 🧠 Interactive Flashcards: Rapid-fire testing on Split Brain, Fencing Tokens, and ZNode mechanics.
  2. 📝 Implementation Cheat Sheet: A comprehensive grid comparing Raft, Paxos, and ZAB for interview readiness.
  3. 🕵️ Hardware Review: A checklist of physical bottlenecks (Network Jitter, Disk IOPS, RAM limits) that break consensus in production.

1. Cheat Sheet: The Big Picture

In Distributed Systems, “Coordination” is the art of getting independent nodes to agree.

Concept The “One Liner” Key Tool/Algo Trade-off
Leader Election “Who is the boss?” Raft, Paxos, Bully Availability (CP system pauses during election).
Split Brain “Two bosses shouting orders.” Quorum (N/2)+1 Requires odd number of nodes (3, 5, 7).
Distributed Lock “Traffic light for resources.” Redlock, ZK Lock TTL vs Correctness (Clock skew risks).
Fencing Token “The Shield against Zombies.” Monotonic ID Requires Storage Layer support.
Consensus “Agreeing on the Log.” Raft, ZAB Latency (Round trips for ACKs).
Service Discovery “Phonebook for dynamic IPs.” ZooKeeper, Etcd Complexity vs Static Config.
Ephemeral Node “I exist only while I am alive.” ZooKeeper Heartbeat traffic overhead.
Linearizability “Real-time strong consistency.” Raft Quorum Read Slower reads (must check with Leader).

2. Decision Tree: Choosing Your Coordination Tool

Not everything needs Raft. Sometimes a TTL is enough.

flowchart TD
    Start[Start: Need to Coordinate?] --> Q1{Is Correctness Critical?}
    Q1 -- No --> Redis[Redis Redlock/TTL - Best for Efficiency]
    Q1 -- Yes --> Q2{Scale of Data?}
    Q2 -- Small --> ZK[ZooKeeper - Best for Service Discovery]
    Q2 -- Large --> Q3{Internal or External?}
    Q3 -- Internal --> Raft[Raft etcd/Consul - Best for Critical Log]
    Q3 -- External --> Spanner[Spanner Paxos - Best for Global SQL]

    style Redis fill:var(--bg-soft),stroke:var(--accent-main),stroke-width:0.125rem,color:var(--text-main)
    style ZK fill:var(--bg-soft),stroke:var(--accent-soft),stroke-width:0.125rem,color:var(--text-main)
    style Raft fill:var(--bg-soft),stroke:var(--text-subtle),stroke-width:0.125rem,color:var(--text-main)
    style Spanner fill:var(--bg-soft),stroke:var(--accent-main),stroke-width:0.125rem,color:var(--text-main)

3. Interactive Flashcards

Test your recall. Click a card to flip it.

What is Split Brain?
A condition where a network partition causes a cluster to split into two independent groups, both potentially electing a leader.
Why do we need Fencing Tokens?
To prevent a "Zombie Leader" (who was paused by GC) from overwriting data written by a new Leader.
What is the Quorum for 5 nodes?
3 nodes. Formula: (N/2) + 1.
What is an Ephemeral Node?
A ZooKeeper node that is automatically deleted when the client session ends (disconnects).
Difference between Raft and Paxos?
Raft is designed for understandability and has a strong Leader. Paxos is the theoretical foundation but is complex to implement.
Why is Redlock controversial?
It relies on system clocks. If a clock jumps, a lock might expire prematurely, violating safety.
What is Linearizability?
Strong consistency guarantee where operations appear to happen instantaneously at some point between their invocation and response.
What is ZAB?
ZooKeeper Atomic Broadcast. It guarantees strict ordering of updates (Primary-Backup model), unlike Raft which is peer-based.

4. Hardware-First Coordination Checklist

Before you rely on a consensus service, verify these physical bounds:

  • Network RTT vs Election Timeout: Is your election timeout set to at least 10x your worst-case network Round Trip Time (RTT)?
  • Disk FSync Latency: Have you benchmarked the fsync() performance on your SSDs? If latency is >1ms, your consensus throughput will be capped.
  • RAM Footprint: Does your Zookeeper data tree fit entirely into physical RAM with 20% headroom for OS Page Cache?
  • Context Switch Rates: If you have high write contention, monitor for CPU “Thrashing”—coordination systems spend significant cycles on locks and context switching.

5. Staff Engineer Challenge: The “Cloud Partition” Paradox

The Scenario: You are running a 5-node Etcd cluster in AWS across 3 Availability Zones (AZs).

  • The Event: AZ-1 and AZ-2 lose a high-speed fiber link. They can both talk to the Internet, but not to each other.
  • The Hardware Impact: One AZ has 2 nodes, another has 2, and the third has 1.
  • The Bug: All nodes are still healthy and “running,” but no one can reach a Quorum (N=3).

The Question:

  1. How do you detect this from a Hardware Networking perspective (rather than just checking app logs)?
  2. Would you manually force a “Single Node Mode” to restore Availability? What is the hardware data corruption risk of doing so?
  3. How would you redesign the physical placement of these nodes to ensure that a single AZ failure NEVER results in a Quorum loss?

6. 🔗 Practice & Next Steps

For high-volume practice and interview preparation, visit the Problem Vault.

You have mastered the art of agreement. Next, we move to Simple Services, where we build real-world systems like ID Generators, Rate Limiters, and Pastebins using these coordination primitives.