How Sagas Survive Even When Compensation Transactions Fail — Completing Distributed Rollbacks with Idempotency and State Machines

When I first introduced the Saga pattern, I had a misconception too. "As long as I implement compensating transactions, failures will be fine." But not long after deploying to production, I encountered this situation: the payment service failed, and while executing the inventory release compensation, the inventory service itself happened to go down. The Saga stopped somewhere in the middle, and after the system restart, there was no way to know how far it had progressed.

A compensating transaction that itself fails is where the real problems begin. A conventional ACID transaction lets the DB engine automatically roll back, but in a distributed environment there is no such safety net. When compensation fails, data inconsistencies become permanently baked in, and recovering from that manually later is genuinely painful.

This article explores how to design and combine two core mechanisms — Idempotency and State Machine — that can drive distributed rollback to completion even when compensating transactions fail. The focus is less on conceptual explanation and more on what code you actually need to write.

Why Compensating Transactions Can't Just Be Retried

The Fundamental Limitations of Saga

As defined on Chris Richardson's microservices.io, a Saga has each service execute its own local transaction and, on failure, reverses the already-completed steps in reverse order. The key point is that a compensating transaction is not about 'restoring data to its previous state' — it is a separate operation that logically undoes the prior action in a business-semantic sense.

'Cancel payment' is not a DELETE of the payment record; it is a new business operation that transitions the status to 'refunded'. This means compensation can itself fail, and the same compensation may even execute twice.

Below is a typical failure scenario that occurs in an e-commerce order Saga.

Two Dimensions of Compensation Failure and the Retry Trap

Compensation failures fall into two broad categories. And separately, if retry logic itself is designed poorly, it creates new problems that weren't there before.

Failing to distinguish between the two means retrying when you should give up, or looping forever when you should stop. And independent of this distinction, if a compensating action is not idempotent, retrying itself creates new bugs. A double refund caused by a retry storm is the classic example. This is why you must establish idempotency before worrying about retry policies.

Idempotency — Making Compensations Safe No Matter How Many Times They Run

Why Idempotency Is Essential

Honestly, I didn't take this part seriously at first. "Compensation will only run once anyway." But when a network response is lost, the orchestrator judges it a failure and retries. If compensation is not idempotent, 'refund $50' executes twice and becomes 'refund $100'.

An Idempotency Key is a value that uniquely identifies each compensation execution. Using the combination saga_id + step_id as the key means that even if the same compensation request arrives N times, it actually executes only the first time.

Implementation Example (Conceptual Example — PostgreSQL-based)

CREATE TABLE idempotency_records (
    idempotency_key VARCHAR(255) PRIMARY KEY,
    saga_id UUID NOT NULL,
    step_id VARCHAR(100) NOT NULL,
    status VARCHAR(50) NOT NULL,  -- 'PROCESSING', 'COMPLETED', 'FAILED'
    result JSONB,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

# Conceptual example (Python pseudo-code)
def execute_compensating_action(saga_id: str, step_id: str, action_fn):
    idempotency_key = f"{saga_id}:{step_id}"
 
    # 1) Short transaction: claim PROCESSING status (unique constraint prevents race conditions)
    with db.transaction():
        row = db.execute(
            """
            INSERT INTO idempotency_records (idempotency_key, saga_id, step_id, status)
            VALUES (%s, %s, %s, 'PROCESSING')
            ON CONFLICT (idempotency_key) DO NOTHING
            RETURNING status
            """,
            [idempotency_key, saga_id, step_id]
        ).fetchone()
 
        if row is None:
            existing = db.execute(
                "SELECT status, result FROM idempotency_records WHERE idempotency_key = %s",
                [idempotency_key]
            ).fetchone()
            if existing["status"] == "COMPLETED":
                return existing["result"]
            if existing["status"] == "PROCESSING":
                raise AlreadyInProgressError(idempotency_key)
            # If FAILED, fall through to allow retry (separate state transition logic)
 
    # 2) External call outside transaction — does not hold DB connection/lock
    try:
        result = action_fn()
    except Exception:
        with db.transaction():
            db.execute(
                "UPDATE idempotency_records SET status='FAILED', updated_at=NOW() WHERE idempotency_key=%s",
                [idempotency_key]
            )
        raise
 
    # 3) Short transaction: finalize result
    with db.transaction():
        db.execute(
            "UPDATE idempotency_records SET status='COMPLETED', result=%s, updated_at=NOW() WHERE idempotency_key=%s",
            [result, idempotency_key]
        )
    return result

There is a tension here that is often missed in practice. The principle that "idempotency check and compensation execution must be a single atomic unit" is correct, but running an external HTTP call inside a DB transaction holds the connection and lock for the entire response wait time, draining the connection pool quickly. That is why the three-step structure — using a unique constraint to prevent contention, claiming the PROCESSING intermediate state first, then performing the external call outside the transaction — is the safe approach. A PROCESSING record that lingers too long is a remnant of a dead worker, so a separate scanner reclaims and redistributes it based on a timeout threshold.

Guaranteeing Receiver-Side Idempotency with the Inbox Pattern

Idempotency keys are needed not only on the sender side but on the receiver side as well. The Inbox pattern ensures that even if the same event is delivered twice from the message broker, it is processed only once.

// Conceptual example (Java — Inbox pattern)
@Transactional
public void handleCompensationEvent(CompensationEvent event) {
    String messageId = event.getMessageId();
 
    // To avoid race conditions, don't use check-then-insert;
    // instead use a messageId unique constraint + duplicate key exception handling.
    try {
        inboxRepository.saveNew(new InboxRecord(messageId, event.getSagaId()));
    } catch (DuplicateKeyException e) {
        log.info("Duplicate message ignored: {}", messageId);
        return;
    }
 
    // Actual compensation logic runs in the same transaction (external calls should be handled in a separate worker)
    inventoryService.releaseReservation(event.getOrderId(), event.getQuantity());
}

existsBy... followed by save is a classic TOCTOU race where two threads can both pass through simultaneously. @Transactional alone won't prevent it — you must combine a unique constraint with duplicate key exception handling.

State Machine — Always Knowing Where the Saga Is

Why Persist State

Idempotency alone is not enough. When the system restarts, there is no way to know "which steps were completed and where compensation should begin." This is why a state machine is needed.

As noted in Dorin Baba's production case study, when a state machine persistently stores state at each step, retrying N times at any step is possible, and even if it fails N-1 times, succeeding on the Nth attempt brings the Saga to normal completion.

Modeling the state transitions of an e-commerce order Saga looks like this. One important point here is separating the terminal state where compensation has completed from the true failure state. Lumping both together as FAILED makes it impossible on the operational dashboard to distinguish 'cases where compensation wrapped up cleanly' from 'cases where data inconsistency remains'.

State Table Design

CREATE TABLE saga_instances (
    saga_id UUID PRIMARY KEY,
    saga_type VARCHAR(100) NOT NULL,
    current_state VARCHAR(100) NOT NULL,
    saga_data JSONB NOT NULL,       -- Saga context (order ID, amount, etc.)
    retry_count INTEGER DEFAULT 0,
    last_updated_at TIMESTAMPTZ DEFAULT NOW(),
    timeout_at TIMESTAMPTZ          -- Deadline for timeout transitions
);
 
CREATE TABLE saga_state_history (
    id BIGSERIAL PRIMARY KEY,
    saga_id UUID NOT NULL REFERENCES saga_instances(saga_id),
    from_state VARCHAR(100),
    to_state VARCHAR(100) NOT NULL,
    event_type VARCHAR(100),
    occurred_at TIMESTAMPTZ DEFAULT NOW()
);

Implementing State Transition Logic (Conceptual Example — Java)

Since the number of transition rules can exceed 10 pairs, initialize with Map.ofEntries() instead of Map.of() (Map.of allows at most 10 pairs).

// Conceptual example (Java)
@Service
public class OrderSagaOrchestrator {
 
    private static final Map<String, List<String>> VALID_TRANSITIONS = Map.ofEntries(
        Map.entry("STARTED", List.of("INVENTORY_RESERVING")),
        Map.entry("INVENTORY_RESERVING", List.of("PAYMENT_PROCESSING", "INVENTORY_COMPENSATING", "FAILED")),
        Map.entry("PAYMENT_PROCESSING", List.of("DELIVERY_SCHEDULING", "INVENTORY_COMPENSATING")),
        Map.entry("DELIVERY_SCHEDULING", List.of("COMPLETED", "PAYMENT_COMPENSATING")),
        Map.entry("PAYMENT_COMPENSATING", List.of("INVENTORY_COMPENSATING", "COMPENSATION_FAILED")),
        Map.entry("INVENTORY_COMPENSATING", List.of("COMPENSATED", "COMPENSATION_FAILED"))
    );
 
    @Transactional
    public void transition(UUID sagaId, String toState, String eventType) {
        SagaInstance saga = sagaRepository.findById(sagaId)
            .orElseThrow(() -> new SagaNotFoundException(sagaId));
 
        String fromState = saga.getCurrentState();
 
        if (!VALID_TRANSITIONS.getOrDefault(fromState, List.of()).contains(toState)) {
            throw new InvalidStateTransitionException(fromState, toState);
        }
 
        saga.setCurrentState(toState);
        saga.setLastUpdatedAt(Instant.now());
        sagaRepository.save(saga);
 
        stateHistoryRepository.save(StateHistory.of(sagaId, fromState, toState, eventType));
 
        triggerNextAction(saga, toState);
    }
}

Preventing Infinite Waits with Timeout Transitions

Another role of the state machine is to provide guardrails that prevent a Saga from getting stuck waiting indefinitely. When a set amount of time passes while waiting for a payment service response, it automatically transitions to the compensation flow.

# Conceptual example — Timeout scanner (Python)
import asyncio
 
async def timeout_scanner():
    while True:
        stale_sagas = db.query("""
            SELECT saga_id, current_state, saga_data
            FROM saga_instances
            WHERE timeout_at < NOW()
              AND current_state NOT IN ('COMPLETED', 'COMPENSATED', 'FAILED', 'COMPENSATION_FAILED')
        """).fetchall()
 
        for saga in stale_sagas:
            await publish_event(
                topic="saga.timeout",
                payload={
                    "saga_id": str(saga["saga_id"]),
                    "timed_out_state": saga["current_state"]
                }
            )
 
        await asyncio.sleep(30)

Hybrid Recovery Strategy — When to Use Retry, Compensation, and DLQ

The Flow That Works in Practice

The retry → DLQ → manual intervention flow is itself a long-established practice. In the context of Saga, it generally converges to the following structure.

1. Transient failure → Retry with exponential backoff + jitter. Retrying all at once without jitter causes a retry storm on downstream services.
2. Retry limit exceeded or permanent failure → Send to DLQ and notify the operations team.
3. DLQ depth > 0 → An alarm signal that data inconsistencies are accumulating.

DLQ depth is a metric that directly shows "the number of events where compensation never reached completion," so it deserves a prominent spot on the operational dashboard. Whether to include it in SLOs is a team-by-team decision, but at minimum you must configure an alert threshold.

Implementing Exponential Backoff + Jitter

Starting from Go 1.21, min is a built-in function and does not need to be defined separately.

// Conceptual example (Go 1.21+)
package saga
 
import (
    "context"
    "math"
    "math/rand"
    "time"
)
 
type RetryConfig struct {
    MaxAttempts int
    BaseDelay   time.Duration
    MaxDelay    time.Duration
}
 
func RetryWithBackoff(ctx context.Context, cfg RetryConfig, fn func() error) error {
    var lastErr error
 
    for attempt := 0; attempt < cfg.MaxAttempts; attempt++ {
        if err := fn(); err == nil {
            return nil
        } else {
            lastErr = err
        }
 
        delay := time.Duration(math.Pow(2, float64(attempt))) * cfg.BaseDelay
        jitter := time.Duration(rand.Int63n(int64(delay/2) + 1))
        sleep := min(delay+jitter, cfg.MaxDelay)
 
        select {
        case <-ctx.Done():
            return ctx.Err()
        case <-time.After(sleep):
        }
    }
 
    return lastErr
}

Improving Trigger Reliability with Transactional Outbox

To prevent situations where a compensation event is published but the local DB commit fails (or vice versa), the Transactional Outbox pattern is needed.

CREATE TABLE saga_outbox (
    id BIGSERIAL PRIMARY KEY,
    saga_id UUID NOT NULL,
    event_type VARCHAR(100) NOT NULL,
    payload JSONB NOT NULL,
    published_at TIMESTAMPTZ,  -- NULL means unpublished
    created_at TIMESTAMPTZ DEFAULT NOW()
);

# Record compensation event in the same local transaction (conceptual example)
with db.transaction():
    db.execute(
        "UPDATE saga_instances SET current_state = 'INVENTORY_COMPENSATING' WHERE saga_id = %s",
        [saga_id]
    )
    db.execute("""
        INSERT INTO saga_outbox (saga_id, event_type, payload)
        VALUES (%s, 'COMPENSATE_INVENTORY', %s)
    """, [saga_id, {"order_id": order_id, "quantity": quantity}])
 
# After transaction commit, the Outbox poller publishes to the message broker.
# Even if the process dies, the event remains in the DB and can be published after restart.

Choosing Tools in Practice

Temporal — The Canonical Approach to Writing Saga as Code

Temporal treats a Saga as a long-running workflow, ensuring that compensation executes with very high reliability even after worker failures (exceptions exist — such as Namespace deletion, loss of the persistence store, or non-deterministic changes to workflow code — so it is not an unconditional guarantee). Because it provides automatic retries, timeout management, and versioning at the platform level, you do not need to build the state machine and retry logic from scratch yourself.

In Temporal, activities must be passed as top-level functions registered with @activity.defn. Workflows assume deterministic replay, so lambdas and closures are not serializable and will fail at runtime.

# Saga workflow using Temporal Python SDK (conceptual example)
from dataclasses import dataclass
from datetime import timedelta
 
from temporalio import workflow, activity
from temporalio.common import RetryPolicy
 
 
@dataclass
class OrderRequest:
    order_id: str
    quantity: int
    amount: int
 
 
@activity.defn
async def reserve_inventory(order: OrderRequest) -> None: ...
 
@activity.defn
async def release_inventory(order: OrderRequest) -> None: ...
 
@activity.defn
async def process_payment(order: OrderRequest) -> None: ...
 
@activity.defn
async def cancel_payment(order: OrderRequest) -> None: ...
 
@activity.defn
async def schedule_delivery(order: OrderRequest) -> None: ...
 
 
@workflow.defn
class OrderSagaWorkflow:
    @workflow.run
    async def run(self, order: OrderRequest) -> str:
        # Compensation stack: store as (activity function, args) tuples for safe serialization and replay.
        compensations: list[tuple] = []
 
        try:
            await workflow.execute_activity(
                reserve_inventory, order,
                start_to_close_timeout=timedelta(seconds=30),
                retry_policy=RetryPolicy(maximum_attempts=3),
            )
            compensations.append((release_inventory, order))
 
            await workflow.execute_activity(
                process_payment, order,
                start_to_close_timeout=timedelta(seconds=60),
                retry_policy=RetryPolicy(maximum_attempts=3),
            )
            compensations.append((cancel_payment, order))
 
            await workflow.execute_activity(
                schedule_delivery, order,
                start_to_close_timeout=timedelta(seconds=30),
            )
            return "ORDER_COMPLETED"
 
        except Exception:
            for comp_fn, comp_arg in reversed(compensations):
                await workflow.execute_activity(
                    comp_fn, comp_arg,
                    start_to_close_timeout=timedelta(seconds=30),
                    retry_policy=RetryPolicy(maximum_attempts=10),
                )
            return "ORDER_COMPENSATED"

Selection Criteria by Framework

Tradeoffs — An Honest Discussion

What Saga Cannot Solve

Some things teams expect when adopting Saga do not hold up in practice.

The Dirty Read problem. Other services can read the intermediate state of a Saga in progress. This means an order might briefly appear as 'payment complete' while payment is still being processed. A separate temporary isolation strategy (e.g., explicitly using a 'reserving' status) is required.

Zero manual intervention does not exist. Saga guarantees eventual consistency, but compensation failures that flow into the DLQ must be handled by the operations team manually. When an external airline API compensation fails in a travel booking system, no matter how well the internal system is built, you still have to coordinate with the airline's own processes.

State Explosion. As the number of states in the state machine grows, design and testing become more complex. In particular, modeling all cases where compensation ordering dependencies exist (a subsequent compensation can only proceed after a prior one completes) quickly makes the state diagram unwieldy.

Saga is not a silver bullet. There are still many situations where Saga does not fit — services built tightly on top of legacy systems, some financial/payment settlement pipelines that must maintain 2PC for regulatory reasons, and cases requiring ultra-low-latency strong consistency. Saga is "a reasonable default in many cases," not an automatic choice.

Conclusion — Where to Start

Saying that the Saga pattern 'solves' the distributed transaction problem is only half right. More precisely, it transfers the complexity of achieving consistency in an environment without automatic rollback into application code and infrastructure.

If you already have a system running Saga, here is the order I recommend for reviewing it.

• Pick one compensating action and call it twice. If the result differs, you need to introduce idempotency keys first. Without that in place, anything you add afterward means retries become bugs.
• If there is no Saga instance table, the next step is the state machine. If you cannot answer "which Saga is this and where is it right now" with a single SQL query after a restart, you cannot begin automating recovery.
• After that comes observability metrics. Specifically, DLQ depth, the number of Sagas that have remained in PROCESSING state beyond a certain time threshold, and the distribution of compensation retry counts — if these three signals are not on the dashboard, data inconsistencies will only be discovered when user complaints start rolling in.

Putting these three things in place in order transforms Saga from 'a system that works fine normally but is unpredictable under failure' into 'a system that eventually self-heals even when failures occur.' In production, that difference shows up quite honestly in the number of pages that go off in the middle of the night.

References

• Microsoft Azure Architecture Center — Compensating Transaction Pattern
• Microsoft Azure Architecture Center — Saga Design Pattern
• microservices.io — Saga Pattern (Chris Richardson)
• microservices.io — Transactional Outbox Pattern
• Temporal — Mastering Saga Patterns for Distributed Transactions in Microservices
• Temporal — Saga Pattern Documentation
• Medium / Dorin Baba — How we used SAGA and State Machine for distributed transactions
• Orkes.io — Compensation Transaction Patterns