The Workflow Survives Server Outages: Stabilizing Payment, Notification, and LLM Pipelines with Temporal.io Durable Execution

Imagine your server crashes immediately after a payment API call completes. Stripe has already processed the charge, but there's no record in your order database. Retrying causes a double charge; not retrying leaves money taken with the order incomplete. When I first encountered this situation, I thought "just use idempotency keys properly" — but in practice, I had to intricately weave together message queues, Redis locks, and separate state tables on top of that. And edge cases always remained in that tangle.

Temporal.io is not a replacement for idempotency keys — it adds a layer above them that persists execution state itself, solving the problem at a different level. It is a Durable Execution platform for reliably handling long-running workflows in distributed systems; workflows resume without interruption even if the server crashes, the network drops, or a deployment causes a restart. Netflix, Stripe, Coinbase, Snap, NVIDIA, and others use it in production — specific adoption context for each case can be found in the official case study resources linked at the bottom of this article.

This article covers how Durable Execution works and how it can be applied to three scenarios — payments, notifications, and LLM pipelines — with TypeScript SDK code. We'll start with the architecture, then move to production code.

Core Concepts

The Problem Temporal Solves

The fundamental problem with conventional distributed workflow processing is that the responsibility for state management falls on the application code. For a three-step process of payment → log entry → email send, you must manually record in the DB how far each step succeeded, then read that state on retry to resume from there. As this accumulates, infrastructure management code starts to outweigh business logic.

Temporal moves this state management responsibility to the platform. Developers express only "what needs to be done" in code, while "how far we've progressed" is automatically recorded in the event history.

Core Components

The Worker's role can be confusing at first, so here's a quick summary.

• The Temporal server (cluster) is only responsible for state storage and orchestration — it does not execute user code.
• A Worker is the user code execution runtime that developers deploy as a separate process. It is bootstrapped with Worker.create({...}) in the service codebase, and can live in the same repository as the API server or be separated into its own deployment unit.
• The API server (client) only requests workflow execution via client.workflow.start() — the actual execution is picked up and handled by a Worker subscribed to the Task Queue.

One key rule worth keeping in mind: Workflow code must be deterministic. Date.now(), Math.random(), and direct external API calls cannot be used inside a Workflow function — they must all be extracted into Activities. I ran into Replay errors early on from doing date calculations inside Workflows; following this rule prevents most issues upfront.

Event History Replay: How It Remembers Where It Left Off

The core of Temporal lies in Event Sourcing. Every state change during Workflow execution — Workflow start, Activity scheduling, Activity completion, Timer firing — is persistently stored as an immutable event log in the cluster.

Worker crash detection is not magic — it is the combination of two timeouts.

• Activity Heartbeat Timeout: If a long-running Activity stops sending periodic heartbeat() calls, the Temporal server detects that the heartbeat interval has been exceeded and marks it as failed.
• StartToClose / ScheduleToClose Timeout: If an Activity fails to report completion within the specified time, it is automatically marked as failed.

In other words, Temporal does not directly observe the death of a Worker process — it determines failure by the fact that "no response arrived within the designated time." Activities marked as failed are rescheduled according to the retry policy, and another Worker picks them up and executes them. At that point, the Workflow replays the event history from the beginning, restoring results of already-completed Activities from the log, and resumes from the incomplete point.

If the payment Activity already succeeded, only the subsequent log-entry Activity is subject to retry. The payment itself is not re-called, so double charging is prevented at the root.

Saga Pattern: Automatic Compensation on Failure

In distributed transactions, when some steps fail, rolling back the already-completed steps in reverse order is the Saga pattern. Temporal lets you express this naturally at the SDK level. The flow in the diagram below is carried directly into the payment example code that follows.

Choosing the Right Tool

Temporal is not the right answer for every situation. Different tools suit different purposes.

For workflows like payments, notifications, and LLM pipelines — where failure is costly and ordering and durability are critical — Temporal is a strong candidate.

Practical Application

Payment Pipeline: Retry Without Double Charging

First, define Activities. All touchpoints with external systems live here, along with the error classes that interact with the retry policy. Note that stripe.charges.create() has been classified as legacy since 2019 — new code should use the PaymentIntents API. Also, in the Stripe SDK, idempotencyKey must be passed as the second argument (options object), not the first argument (parameters object), for it to actually take effect.

// activities.ts
import Stripe from 'stripe';
 
const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);
 
export class PaymentDeclinedError extends Error {
  constructor(message: string) {
    super(message);
    this.name = 'PaymentDeclinedError';
  }
}
 
export async function chargePayment(
  orderId: string,
  amount: number,
): Promise<string> {
  try {
    const intent = await stripe.paymentIntents.create(
      {
        amount,
        currency: 'krw',
        confirm: true,
        payment_method: await resolvePaymentMethod(orderId),
        metadata: { orderId },
      },
      { idempotencyKey: `charge-${orderId}` },
    );
    return intent.id;
  } catch (err) {
    if (err instanceof Stripe.errors.StripeCardError) {
      throw new PaymentDeclinedError(err.message);
    }
    throw err;
  }
}
 
export async function reserveInventory(orderId: string): Promise<string> {
  return inventoryService.reserve(orderId);
}
 
export async function releaseInventory(reservationId: string): Promise<void> {
  await inventoryService.release(reservationId);
}
 
export async function registerShipment(orderId: string): Promise<string> {
  return shippingService.register(orderId);
}
 
export async function refundPayment(chargeId: string): Promise<void> {
  await stripe.refunds.create({ payment_intent: chargeId });
}

The Workflow code expresses only the pure flow. The proxyActivities options may be unfamiliar, so here are the key ones:

• startToCloseTimeout: The maximum time a single execution of an Activity must complete within. Exceeding it triggers the retry policy.
• retry.maximumAttempts: The upper limit on retry attempts.
• retry.backoffCoefficient: The multiplier for increasing wait time between retries. A value of 2 produces 1s → 2s → 4s growth.
• retry.nonRetryableErrorTypes: If error.name is in this list, the Activity is immediately marked as failed without retry. Use this for errors like PaymentDeclinedError where retrying would yield the same result.

// workflows.ts
import { proxyActivities, ApplicationFailure } from '@temporalio/workflow';
import type * as activities from './activities';
 
const acts = proxyActivities<typeof activities>({
  startToCloseTimeout: '30 seconds',
  retry: {
    maximumAttempts: 3,
    backoffCoefficient: 2,
    nonRetryableErrorTypes: ['PaymentDeclinedError'],
  },
});
 
export async function orderPaymentWorkflow(params: {
  orderId: string;
  amount: number;
}): Promise<void> {
  const compensations: Array<() => Promise<void>> = [];
 
  try {
    const chargeId = await acts.chargePayment(params.orderId, params.amount);
    compensations.push(() => acts.refundPayment(chargeId));
 
    const reservationId = await acts.reserveInventory(params.orderId);
    compensations.push(() => acts.releaseInventory(reservationId));
 
    await acts.registerShipment(params.orderId);
  } catch (err) {
    for (const compensate of compensations.reverse()) {
      await compensate();
    }
    throw err;
  }
}

There is one design point that is easy to miss here. You should not route every error directly to compensation. For example, if registerShipment fails due to a transient network error, that is the job of the Activity retry policy — Saga compensation should only trigger if the Activity still fails after all retries are exhausted. Temporal's Activity retry is already configured in the proxyActivities options above, and the exception only propagates to the Workflow after maximumAttempts is exhausted. Conversely, for errors where retrying is meaningless — like "payment succeeded but shipment is fundamentally impossible" — the Activity should throw ApplicationFailure.nonRetryable(...) internally to immediately enter the compensation flow. Separating recoverable-by-retry errors from recoverable-only-by-compensation errors at the Activity level is the practical core of Saga design.

On the client side, specifying payment-${orderId} as the Workflow ID prevents duplicate workflow executions for the same order entirely.

// client.ts
import { Client } from '@temporalio/client';
import { orderPaymentWorkflow } from './workflows';
 
const client = new Client();
 
const handle = await client.workflow.start(orderPaymentWorkflow, {
  args: [{ orderId, amount }],
  taskQueue: 'payment-queue',
  workflowId: `payment-${orderId}`,
});

It's also worth noting that Temporal's workflowId deduplication and Stripe's idempotencyKey do not replace each other. The workflowId prevents duplicate workflow executions, while the Stripe idempotency key protects against Stripe processing the same charge twice when an Activity is retried. Both layers must be in place for double-charge risk to be practically eliminated.

Notification Pipeline: Guaranteed Ordering and Durable Timers

sleep is one of Temporal's core features. Timer state is preserved even when a Worker restarts, so in-memory setTimeout calls can never be lost in a deployment.

// workflows.ts
import { proxyActivities, sleep } from '@temporalio/workflow';
import type * as activities from './activities';
 
const acts = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',
  retry: { maximumAttempts: 5 },
});
 
export async function orderNotificationWorkflow(params: {
  userId: string;
  orderId: string;
}): Promise<void> {
  await acts.sendPushNotification(params.userId, 'Your order has been received');
 
  await sleep('5 minutes');
  await acts.sendEmailNotification(params.userId, params.orderId);
 
  await sleep('24 hours');
  await acts.sendShippingUpdateNotification(params.userId, params.orderId);
}

Using Signal, you can control the flow of a running Workflow with external events. This is particularly useful for implementing human-in-the-loop approval waits.

import { defineSignal, setHandler, condition } from '@temporalio/workflow';
 
const approvalSignal = defineSignal<[boolean]>('approval');
 
export async function approvalWorkflow(requestId: string): Promise<string> {
  let approved = false;
  let received = false;
 
  setHandler(approvalSignal, (isApproved: boolean) => {
    approved = isApproved;
    received = true;
  });
 
  await acts.notifyApprover(requestId);
 
  const didReceive = await condition(() => received, '72 hours');
 
  if (!didReceive) return 'timeout';
  return approved ? 'approved' : 'rejected';
}

LLM Pipeline: Preventing Re-billing on Expensive Calls

LLM API calls are costly and have non-zero failure rates. In Temporal, a failure midway through will never cause already-completed LLM calls to re-execute. Activities that already succeeded are simply retrieved from the event history — they are not re-run.

// activities.ts
export async function splitDocument(documentId: string): Promise<string[]> {
  const document = await storage.get(documentId);
  return chunkText(document, 512);
}
 
export async function generateEmbedding(chunk: string): Promise<number[]> {
  const response = await openai.embeddings.create({
    model: 'text-embedding-3-small',
    input: chunk,
  });
  return response.data[0].embedding;
}
 
export async function summarizeChunks(chunks: string[]): Promise<string> {
  const response = await openai.chat.completions.create({
    model: 'gpt-4o',
    messages: [
      { role: 'system', content: 'Summarize the given paragraphs into a single paragraph.' },
      { role: 'user', content: chunks.join('\n\n---\n\n') },
    ],
  });
  return response.choices[0].message.content ?? '';
}
 
export async function storeEmbeddings(
  documentId: string,
  embeddings: number[][],
): Promise<void> {
  await vectorDB.upsert({ id: documentId, embeddings });
}
 
export async function storeResult(
  documentId: string,
  summary: string,
): Promise<void> {
  await db.summaries.insert({ documentId, summary });
}

In the Workflow, chunks are passed as actual prompts, and parallel embedding generation should be considered alongside concurrency limits. Leaving Promise.all(chunks.map(...)) as-is schedules as many Activities as there are chunks simultaneously, making it easy to hit OpenAI rate limits. In practice, two common buffers are used:

• maxConcurrentActivityTaskExecutions on the Task Queue Worker: Limits the number of concurrently executing Activities per Worker.
• Application-level batching or semaphores: Slice chunks into batches for sequential processing, or use a limiter like p-limit, as shown below.

// workflows.ts
export async function documentProcessingWorkflow(params: {
  documentId: string;
}): Promise<string> {
  const chunks = await acts.splitDocument(params.documentId);
 
  const batchSize = 8;
  const embeddings: number[][] = [];
  for (let i = 0; i < chunks.length; i += batchSize) {
    const batch = chunks.slice(i, i + batchSize);
    const batchEmbeddings = await Promise.all(
      batch.map((chunk) => acts.generateEmbedding(chunk)),
    );
    embeddings.push(...batchEmbeddings);
  }
  await acts.storeEmbeddings(params.documentId, embeddings);
 
  const summary = await acts.summarizeChunks(chunks);
 
  await acts.storeResult(params.documentId, summary);
 
  return summary;
}

Even if storeResult fails after summarizeChunks succeeds, the summarization call is not re-executed. The already-completed result is fetched from the event history and only storeResult is retried. This means you never pay again for calls that can cost anywhere from a few cents to several dollars.

For cases where LLM token streaming is needed, the Temporal team has been sharing streaming use cases based on the Signal & Update primitives. The specific API and official documentation location are best confirmed in the official blog and release notes linked in the references below.

Pros and Cons

Advantages

Disadvantages and Considerations

Common Mistakes in Practice

1. Using non-deterministic code inside Workflow code

// Wrong — produces different values on Replay, causing errors
export async function myWorkflow() {
  const timestamp = Date.now();
  const value = Math.random();
}
 
// Correct — handle in an Activity
export async function myWorkflow() {
  const timestamp = await acts.getCurrentTimestamp();
}

2. Deploying Workflow code changes without versioning

Changing logic inside a Workflow function while Workflows are still running will cause Replay to fail. You need a strategy: use the patched() API to distinguish between old and new versions, or isolate execution with Worker Versioning before rolling out.

3. Unbounded event history growth

A single Workflow accumulating tens of thousands of events affects performance. For high-iteration workloads, using continueAsNew to periodically reset the event history is recommended.

import { continueAsNew } from '@temporalio/workflow';
 
export async function longRunningWorkflow(iteration: number): Promise<void> {
  await acts.processNextBatch(iteration);
 
  if (iteration < 10000) {
    await continueAsNew<typeof longRunningWorkflow>(iteration + 1);
  }
}

Closing Thoughts

Temporal offers a shift in thinking for distributed systems: "the platform, not the application code, is responsible for state management." Problems like double payment charges, missed notifications, and wasted LLM calls are resolved at the infrastructure level, freeing developers to focus on business logic.

A natural first reaction is "can't we just use queues properly?" But once requirements like Saga compensation, long-running timers, human-in-the-loop approval waits, and partial retries in LLM pipelines start stacking up, the state tables and retry logic built on top of queues get complex fast. That is the point where Temporal's abstraction actually reduces lines of code.

If you're considering adoption now, here is the order I'd recommend. Each step is a minimum validation checkpoint to reduce risk for the next.

1. Set up a local Temporal cluster to establish a development loop: Clone the official temporalio/docker-compose repo and start with docker compose up. The built-in UI (localhost:8080) lets you visually inspect event history and failure reasons, significantly reducing the time needed to learn the SDK syntax.
2. Migrate one simple async task from your existing codebase to an Activity first: Start with something where the retry value on failure is clear, like email sends or external API calls. Observe Replay and retry behavior in an isolated scope before expanding — this minimizes mistakes like determinism rule violations in later migrations.
3. Run a production pilot on Temporal Cloud's free tier: Validate workflowId-based deduplication and automatic retries against real traffic without the burden of self-hosting. Once you pass this point, you can make operational decisions about self-hosting and cluster sizing based on actual data.

Temporal continues to expand SDK integrations and streaming/external storage features in the AI agent orchestration space as well. For the latest roadmap and detailed specs, checking the official blog and release notes directly is your most reliable source.

References

• Temporal Official Docs — Understanding Durable Execution
• Temporal Official Blog — What is Durable Execution
• Temporal Official Docs — Use Cases and Design Patterns
• Temporal Official Blog — AI/ML and Data Engineering Workflows
• Temporal Official Blog — Saga Pattern Made Easy
• Temporal Case Study — Coinbase
• Temporal Official — Stripe Case
• Temporal Official — Netflix Case
• Stripe Official Docs — PaymentIntents API
• Stripe Official Docs — Idempotent Requests
• GitHub — Temporal TypeScript Samples
• GitHub — Temporal Go Samples
• Medium — Temporal in AI Workflows: Building Reliable LLM Pipelines
• DZone — Temporal Workflow Design Patterns
• HackerNoon — A Practical Guide to Temporal: What It Does, How It Compares, and When to Use It