Unleashing the Power of Idempotency: Building Reliable and Consistent API Operations

What is Idempotency?

An operation is considered idempotent if performing it multiple times yields the same result as if it was done only once. So, an idempotent API is one where multiple identical requests have the same effect as a single request. An idempotent API operation guarantees that no matter how many times you make the same request, the system will always end up in the same state. This property is particularly important in distributed systems where network issues, timeouts or failures can cause requests to be retried or duplicated. Idempotent APIs help ensure predictable and reliable behaviour. It allows developers to retry or repeat API requests without fear of unintended side effects or duplicated actions.

Let us understand it with an elementary mathematics example – Adding 5 to a non-zero number is not an idempotent operation as this will give different results depending on the number of times this operation is done. However, adding 0 to a number is an idempotent operation as no matter how many times we redo this operation, the result will be the same.

Why is it important?

When dealing with distributed systems, there is always the possibility of things going wrong, such as connections could drop midway or requests can timeout. In such cases, it is very likely for clients to retry. When requests are retried, non-idempotent APIs can cause significant side-effects by creating additional resources or changing them unexpectedly which will cause a lot of inconsistency in the system. This can be even more dangerous when the accuracy of the data is very important. Having idempotent APIs enables clients to put retry mechanisms, which not just makes the system more fault-tolerant, but also provides a smoother and more seamless user experience. 

Let’s take the example of a payment system and see how idempotency plays an important role in the reliability and consistency of the system and what would happen if our system does not support idempotency.

Image Source: ResearchGate

In a payments system, there are various components and systems at play.

1. First, there is the merchant application (client) through which the user initiates the payment request. It communicates with the payment gateway to submit payment details and initiate the transaction.
2. Once the payment request is initiated, it goes through a payment gateway that acts as a middleware between the merchant’s application and the payment processors. It receives payment requests from the merchant and forwards them to the appropriate payment processor. The gateway is responsible for coordinating communication between different systems.
3. The payment processor is responsible for securely processing the payment transactions. It interacts with various financial networks, such as credit card networks or banking systems, to authorise and settle transactions.
4. At last a transaction database is used to store and persist payment transaction details. It serves as the system of record for completed payments, ensuring data integrity and auditability.
5. Finally, the appropriate response is sent back to the merchant application through the payment gateway.

Suppose a user initiates a transaction. The request is forwarded by the gateway to the payment processor, which gets successfully processed and stored as a record in the database. Now the response is sent to the payment gateway but because of communication failure, the client (merchant app)  does not receive the payment success response. The client does not see it as a successful transaction. If the client retries to make the payment, which most certainly they will, a non-idempotent system will result in a duplicate payment as the retry requests will be processed independently.

This can cause a lot of issues as it will result in financial loss and also customer dissatisfaction discouraging the user from doing online transactions. However, if we were using idempotent APIs, doing a retry will not actually process the payment again and instead return the successful response for the previous request. So we are saved from the duplicate transaction. So. making the APIs idempotent can help make a system more reliable, resilient and fault-tolerant.

Rest APIs and Idempotency

POST: Generally, POST APIs are used to create a new resource on the server. So if we execute the same POST request N times, we will have N new resources created on the server. So, it is not idempotent by default.

GET, HEAD, OPTIONS, TRACE: These HTTP methods never change the resource state of the system. They are purely for retrieving the resource data. Hence, they are idempotent.

PUT: PUT APIs are used to update a resource state. So if we make a PUT request N times, the first request will update the resource and the subsequent N-1 requests will overwrite the same resource without making any effective change. This makes it idempotent.

DELETE: Similar to PUT request, when we make a DELETE request N number of times, the first request will delete the requested resource and the response code will be 200. The following requests will respond with 204 making no change in the state of the system. Hence these requests will behave as idempotent.

Note: The actual idempotency of these methods depends on their specific implementation. While the HTTP specifications define the expected behaviour of each method, it’s ultimately up to the server-side implementation to adhere to those specifications and guarantee idempotency. Different server implementations may have variations in how they handle idempotency, so it’s important to consider the implementation details and documentation of the specific REST API you are working with.

Ways To Achieve

Idempotent Keys: Clients can provide an idempotency key in the API request header or parameter. This can be used to detect duplicate requests and ensure that subsequent requests do not cause any unintended side effects.

In the example taken above, the client can pass a unique identifier in the header or in a param that can be used to identify duplicate requests. We can keep track of the unique identifier key that is being passed with the requests. When we get a new request, we look for the key in our data store. If we don’t find a match of the unique key we process the request and return the appropriate response and store it in a cache. In case we do find it, we return the cached response. This will save us from doing the same operation multiple times.

For example, let’s consider an API endpoint for creating a new user. The client can include an idempotency key in the request header:

POST /users
Content-Type: application/json
Idempotency-Key: unique-key-123

{
  "name": "John Doe",
  "email": "johndoe@example.com"
}
Code language: JavaScript (javascript)

The server can store the idempotency key in a database or cache and associate it with the request. When a new request is received, the server checks if the idempotency key has been used before. If it is a duplicate request, the server can simply return the cached response without processing the request again.

Transactional Operations: If a database operation involves multiple steps or actions, we should try making it transactional. Transactions ensure that either all the steps are successfully completed or none of them is. By using transactional database operations, we can prevent inconsistent states caused by the partial execution of requests.

For example, let’s consider an API endpoint for transferring funds between two bank accounts. The server can use a transaction to ensure that the transfer is processed atomically:

POST /transfers
Content-Type: application/json

{
  "from_account": "account-123",
  "to_account": "account-456",
  "amount": 100.00
}
Code language: JavaScript (javascript)

@transaction.atomic
def transfer_funds(from_account, to_account, amount):
    # Perform necessary steps for fund transfer
    # ...

    # Update account balances
    # ...

    # Log the transaction
    # ...
Code language: PHP (php)

By using a transaction, if the transfer operation is retried, the server ensures that either the entire transaction is successfully completed (including updating balances and logging), or none of it is executed, preventing duplicate transfers or inconsistent account balances.

Idempotent API Design: We can use a hash function that takes the API parameters as input and generates a unique hash value. We can keep track of these hash values and use them to identify duplicate requests. Whenever there is a new request we create a hash using the parameters of the request and look for the hash in our database. If we don’t find the hash it means the request is new and we need to return the appropriate response after processing it. We will also need to cache the response. In case we do find the hash we simply return the cached response identifying it as a duplicate request.

For example, let’s consider an API endpoint for creating a new blog post. The server can generate a unique hash based on the content of the blog post and use it to identify duplicate requests:

POST /posts
Content-Type: application/json

{
  "title": "My Blog Post",
  "content": "Lorem ipsum dolor sit amet, consectetur adipiscing elit."
}
Code language: JavaScript (javascript)

In the server-side code, the server can generate a hash of the blog post content and check if it has been processed before:

import hashlib

def create_post(request):
    title = request.data['title']
    content = request.data['content']

    # Generate hash of the content
    content_hash = hashlib.sha256(content.encode()).hexdigest()

    # Check if the hash exists in the database or cache
    if check_duplicate(content_hash):
        # Return cached response for duplicate request
        return get_cached_response(content_hash)
    else:
        # Process the request and generate response
        response = process_post_creation(title, content)

        # Cache the response with the content hash
        cache_response(content_hash, response)

        return response
Code language: PHP (php)

Conclusion

By embracing idempotency in API design, developers can create fault-tolerant systems that provide smoother user experiences, prevent data inconsistencies, and handle retries seamlessly. It is a crucial consideration in building robust and reliable API architectures that stand up to the challenges of today’s complex and interconnected environments.

If you would like to explore more good coding practices, check out our post on Mastering SOLID Principles: 5 Points Guide to Writing High-Quality Maintainable Code.