Distributed Locks Explained

Quick answer

A distributed lock coordinates access to a shared resource across multiple processes, servers, or services. It tries to ensure that only one actor performs a critical operation at a time.

Distributed locks can be useful, but they are easy to misuse. Network partitions, process pauses, expired leases, retries, and clock assumptions can make locking behavior surprising.

Why distributed locks exist

In a single process, a local lock can protect shared memory. In a distributed backend, work may run on many instances:

worker A
worker B
worker C

If all workers can process the same account, rebuild the same report, or run the same scheduled job, the system may need coordination.

A distributed lock is one way to say: “Only one worker should do this specific work right now.”

Common use cases

Distributed locks are often considered for:

• Running one scheduled job across many instances.
• Preventing duplicate expensive report generation.
• Coordinating a leader process.
• Protecting rare administrative operations.
• Serializing work for one tenant or resource.

But many duplicate-work problems are better solved with idempotency, unique constraints, queues, or optimistic locking.

Lease-based locks

Many distributed locks use leases. A worker acquires a lock with an expiration time.

lock key: rebuild-report:account-42
owner: worker-7
expires in: 30 seconds

The expiration prevents a lock from being held forever if the owner crashes. But it also creates risk: if the owner is slow or paused and the lease expires, another worker may acquire the lock while the first worker is still running.

This is why lock duration, renewal, and operation time matter.

Fencing tokens

A fencing token is a monotonically increasing number issued when a lock is acquired.

worker A gets token 101
worker B later gets token 102

The protected resource accepts writes only from the newest valid token. If worker A resumes late and tries to write with token 101, the resource rejects it because token 102 already exists.

Fencing tokens are a common safeguard when lock expiration can allow old owners to continue running.

Alternatives to distributed locks

Before adding a distributed lock, consider simpler or safer patterns.

Problem	Alternative
Duplicate API retry	Idempotency key.
Concurrent row update	Optimistic locking.
Duplicate job enqueue	Unique database constraint.
Competing workers	Queue with visibility timeout and idempotent consumer.
One scheduled task	Platform scheduler or leader election with clear leases.

Read Idempotency in APIs Explained and Optimistic Locking Explained before choosing a lock.

Common failure modes

Distributed locks can fail in non-obvious ways:

• The lock holder pauses for garbage collection.
• The lock service is reachable by one node but not another.
• A lease expires while work is still running.
• A retry acquires a second lock after an uncertain response.
• The system assumes clocks are perfectly synchronized.
• The protected resource does not verify ownership.

These are not theoretical concerns. They show up during outages and high load.

When a lock is reasonable

A distributed lock may be reasonable when:

• The operation is rare and expensive.
• Duplicate work is wasteful but not catastrophic.
• The lock has a short lease.
• The operation can tolerate retry or repair.
• The protected write uses fencing or another validation layer.
• The team can monitor lock acquisition, hold time, and failures.

Locks are much riskier when they protect money movement, inventory correctness, or security decisions without additional safeguards.

Common mistakes

The first mistake is assuming a distributed lock gives perfect mutual exclusion under all failures. It usually gives a best-effort lease with important conditions.

The second mistake is holding the lock while doing slow network calls. The longer the critical section, the more ways it can fail.

The third mistake is not making the operation idempotent. Even with a lock, duplicate execution can happen.

The fourth mistake is not recording enough ownership metadata to debug stuck or expired locks.

Practical checklist

Before using a distributed lock:

• Define the exact resource being protected.
• Keep the critical section small.
• Use lease expiration and renewal carefully.
• Add fencing tokens when stale owners can cause harm.
• Make the operation idempotent when possible.
• Monitor lock wait time, hold time, and failure rate.
• Document what happens after lock service failure.

Related reading

Read Optimistic Locking Explained for row-level conflict detection and Message Queues Explained for competing consumer workflows. For duplicate-safe API operations, read Idempotency in APIs Explained. You can also browse the System Design topic cluster.