Database Replication and Replication Lag
How leader-follower replication scales reads and survives failures, and why async followers serve stale data: with 0.2 s lag, 40% of reads within half a second of a write were stale.
Database replication keeps copies of data on several machines for high availability, read scaling and locality. In leader-follower setups all writes go to the leader and followers apply its replication log. Interviewers expect the trade-off: asynchronous replication is fast but followers lag, so reads can be stale and a failover can lose recent acknowledged writes.
Replication keeps copies of a database on several machines for availability and read scaling. In leader-follower setups writes go to the leader and flow to followers. With asynchronous replication, followers lag, so a user who writes then reads from a follower may not see their own change: in a simple model with 0.2 s lag, a read at a random moment within 0.5 s of a write is stale 40% of the time (0.2 / 0.5).
- Database replication provides high availability, read scaling and geographic proximity, but it does not scale writes, because every write still goes through the leader.
- In single-leader replication, PostgreSQL streams its write-ahead log to standbys, MySQL ships its binary log, and MongoDB replicas apply an operation log.
- Synchronous replication reports a commit only after a follower has stored it, while asynchronous replication replies at once and can lose acknowledged writes if the leader crashes.
- In a simulation with a fixed 0.2 second replication lag and reads within 0.5 seconds of a write, about 40% of follower reads were stale, matching 0.2 / 0.5.
- Read-your-writes consistency can be provided by reading from the leader after a write, or by having the follower wait until it has applied the write's log position.
- Split brain is when two nodes both believe they are the leader and accept conflicting writes, so failover must fence the old leader.
01Why replicate
A single database server is a single point of failure, and one machine can only serve so many reads. Replication keeps copies of the data on several servers. If the primary fails, another can take over (high availability). Read traffic can go to the copies (read scaling), which matters because most applications read far more than they write. And copies can sit in different regions, so reads are near users and one data centre outage does not lose data.
Replication does not scale writes. In the common design, every write still goes to one machine. Scaling writes needs partitioning (sharding), a separate topic.
02Leader-follower
In single-leader replication, one node (the leader or primary) accepts all writes. It records each change in a replication log, and followers (replicas, secondaries) apply the same changes in the same order. Clients read from the leader or from any follower. PostgreSQL streams its write-ahead log to standbys, MySQL ships its binary log, and MongoDB replicas apply an operation log.
Multi-leader and leaderless designs allow writes in more places, useful across regions or for offline clients, but need conflict resolution (see the CAP and quorums topic). Single leader is simplest and the most common default.
03Synchronous versus asynchronous
Synchronous replication: the leader reports a commit only after at least one follower has stored it. No acknowledged write is lost if the leader dies, but every write waits for the network and the slowest involved follower, and if that follower is down, writes stall. Asynchronous replication: the leader commits locally and replies at once, sending changes in the background. Writes are fast and not blocked by followers, but a leader crash can lose acknowledged writes that had not yet reached any follower.
A middle road is semi-synchronous: wait for one follower and replicate asynchronously to the rest, so you keep a recent copy somewhere without being at the mercy of all of them. Many managed databases offer this.
04Replication lag
With asynchronous replication, a follower is always some amount behind, from milliseconds to many seconds or minutes under load. A user who writes and then reads from a follower can miss their own change. Model it: writes arrive at random times, the follower applies each one after a lag of 0.2 seconds, and the user reads from the follower at a random point within 0.5 seconds after writing.
import random
stale = 0
for _ in range(10000):
tw = random.random() * 100 # time of the write
tr = tw + random.random() * 0.5 # read within half a second
stale += tr < tw + 0.2 # follower not caught up yet
print(stale / 10000) # 0.398About 40% of those reads were stale, close to the 0.2 / 0.5 you would expect. If your page saves a profile and immediately reloads it from a replica, two out of five users would see the old data. This simulation uses a fixed 0.2 s lag. Real lag varies and spikes under heavy writes or long-running queries on the follower.
05Anomalies and fixes
Three typical anomalies. Read-your-writes: you update something and then do not see it. Monotonic reads: you see a new value on one request and an older value on the next, because the two reads hit different followers. Causality: you see an answer before the question it responds to, because they were applied on different shards or in different orders.
Fixes: after a write, read from the leader for a short window or for that user's own data. Pin each user's session to one follower to avoid going backwards. Pass a log position (a token) from the write to the next read, and have the follower wait until it has applied that position. These add some complexity, so decide which screens truly need fresh data. A list of trending items can lag a few seconds. An order confirmation page cannot.
06Failover
When the leader dies, a follower must be promoted. Steps: detect the failure (usually by missed heartbeats and timeouts), choose the most up-to-date follower, reconfigure clients and other followers to use it, and make sure the old leader cannot come back and accept writes as well (split brain). With asynchronous replication, the new leader may lack the final writes of the old one. Those are lost, or conflict if the old leader returns.
Set failure timeouts carefully: too short and a brief network glitch triggers an unnecessary failover, too long and recovery is slow. Use a consensus-based coordinator (Raft, ZooKeeper-style) to elect leaders safely, and rehearse failover regularly.
07Common interview questions
Q: Why replicate a database?
A: High availability, read scaling and geographic proximity. It does not scale writes.
Q: Synchronous versus asynchronous replication?
A: Synchronous waits for followers and loses nothing but is slower and can stall. Asynchronous is fast but can lose recent writes on failover and serves stale reads.
Q: What is replication lag and how does it show up?
A: The delay before followers apply changes. Users see missing recent writes or time going backwards.
Q: How do you provide read-your-writes consistency?
A: Read from the leader after a write, or wait for the follower to reach the write's log position.
Q: What is split brain?
A: Two nodes both believing they are the leader, accepting conflicting writes.
Q: Does replication help with write throughput?
A: No. All writes still go through the leader; use sharding for that.