3 min read

Deep Dive: Designing Idempotent Systems

In distributed systems, failures are not just possible; they are guaranteed. Network partitions, timeouts, and process crashes make it essential to design systems that can safely retry operations. This is where idempotency becomes crucial.

What is Idempotency?

In mathematics and computer science, an operation is idempotent if applying it multiple times yields the same result as applying it once. In the context of API design and distributed systems, it means that a client can safely retry a request without causing unintended side effects (like double-charging a credit card or creating duplicate database records).

HTTP Methods and Idempotency

  • GET, PUT, DELETE, HEAD, OPTIONS: Inherently idempotent by design.
  • POST: Not idempotent. A repeated POST request typically results in multiple resource creations.

Why is Idempotency Hard?

The challenge lies in the unpredictable nature of networks. When a client sends a request and receives a timeout, it falls into a state of uncertainty:

  1. Did the request fail before reaching the server?
  2. Did the server process the request, but the response was lost?
  3. Is the server still processing the request but taking too long?

If the client retries the request (which it should, to ensure delivery), the server must be able to recognize the retry and handle it appropriately.

Designing an Idempotent API

To make operations idempotent, especially those like POST, we introduce an Idempotency Key.

The Idempotency Key

An idempotency key is a unique identifier generated by the client and sent along with the request (usually in a header like Idempotency-Key).

Workflow:

  1. Client Generation: The client generates a UUID for the operation.
  2. First Request: The client sends the request with the Idempotency-Key.
  3. Server Processing:
    • The server checks if the key exists in its datastore.
    • If not, it saves the key (often in a “started” state), processes the request, saves the response, updates the key state to “completed”, and returns the response.
    • If the key exists and the state is “completed”, the server simply returns the saved response without reprocessing.
    • If the key exists and the state is “started”, the server knows a concurrent request is happening and can return an error (e.g., 409 Conflict).

The Database Side

Storing the idempotency state safely requires careful consideration of isolation levels and transactions.

CREATE TABLE idempotency_keys (
    idempotency_key VARCHAR(255) PRIMARY KEY,
    user_id UUID NOT NULL,
    request_path VARCHAR(255) NOT NULL,
    request_params JSONB NOT NULL,
    response_code INT,
    response_body JSONB,
    status VARCHAR(50) NOT NULL, -- 'started', 'completed'
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

When implementing the check-and-set logic, database constraints (like a Unique Constraint on idempotency_key and user_id) ensure that race conditions don’t lead to duplicate processing.

Best Practices

  1. Scope by User/Tenant: Idempotency keys should be scoped to a specific user to prevent accidental collisions.
  2. Expiration: Keys should expire after a reasonable amount of time (e.g., 24 hours). This prevents the database from growing indefinitely.
  3. Hash the Request Payload: To ensure a client doesn’t reuse an idempotency key for a completely different request, hash the request payload and compare it.
  4. Graceful Error Handling: If the client changes the payload for the same key, return a 400 Bad Request.

Conclusion

Idempotency is a foundational pattern for building reliable distributed systems. By implementing idempotency keys and ensuring safe retries, we protect our systems from the chaos of network unreliability and provide a robust experience for our users.

Graph View