Your server can die mid-function and Temporal will still finish the job

Uber published a number in June 2023 that I had to read twice. Their Cadence workflow engine was running more than 12 billion workflow executions and 270 billion actions a month, across over a thousand internal services. Not five-second background jobs either. Long-running orchestrations, batch pipelines, distributed cron, model training, business processes that stay alive for hours or months.

The scale is impressive, but the scale is not the part worth stealing. The programming model is.

Normally, when you need a process that survives restarts, you build a state machine by hand: a status column, a retry counter, a next_retry_at timestamp, a cron job that sweeps stuck rows, and a recovery script nobody has run since the person who wrote it left. With a durable execution engine you write something that reads like an ordinary Go function.

go
Copy
1func OrderWorkflow(ctx workflow.Context, order Order) error {
2    if err := chargeCustomer(ctx, order); err != nil {
3        return err
4    }
5
6    if err := createShipment(ctx, order); err != nil {
7        return err
8    }
9
10    if err := workflow.Sleep(ctx, 24*time.Hour); err != nil {
11        return err
12    }
13
14    return notifyCustomer(ctx, order)
15}

The process running that function is allowed to vanish. Evict the pod, restart the node, ship a deploy, scale the worker deployment to zero for a week. When a compatible worker comes back, execution picks up with its previous progress intact.

That property is called durable execution, and most short explanations of it are wrong in a way that matters.

You will read that Temporal persists the workflow's stack, variables and timers. That description gets the developer experience across, but it is not what the system does. Nothing is dumping your worker's RAM or serialising a native Go call stack.

What the Temporal Service stores is an append-only event history: the decisions the workflow made and the results it received. For an order, that history holds something conceptually like this.

Copy
Workflow started: order-481
ChargeCustomer activity scheduled
ChargeCustomer activity started
ChargeCustomer activity completed: payment-913
Shipment activity scheduled
Shipment activity completed: shipment-772
Timer started: 24 hours
Timer fired
NotifyCustomer activity scheduled
NotifyCustomer activity completed
Workflow completed

That history lives in Temporal's database. Your workers stay separate processes in your own infrastructure, polling task queues, running your workflow and activity code, and reporting commands and results back.

When a worker picks up a workflow task, the SDK loads the event history, starts your workflow function from line one, replays the recorded events against it, rebuilds your local variables and control flow as it goes, and keeps going until it hits work that has not been recorded yet. That is the whole trick. The code appears to resume where it stopped because it actually ran again from the top inside a different process, fed its own past.

Say the customer has already been charged. On replay, the workflow reaches this line again:

go
Copy
1workflow.ExecuteActivity(ctx, ChargeCustomer, order)

Temporal finds a completion event for that activity in the history and hands back the recorded result. Nobody gets charged twice. The workflow moves on to the first step that has no recorded outcome.

The cost of this is determinism. Given the same history, your workflow code has to produce the same sequence of Temporal commands, which rules out a familiar set of calls inside workflow functions:

go
Copy
1// Not inside a Workflow.
2time.Now()
3rand.Intn(100)
4http.Get("https://payment-service")
5database.Exec(...)
6go someFunction()

Anything touching the outside world moves into an Activity. Anything affecting control flow goes through a Temporal-aware API:

go
Copy
1workflow.Now(ctx)
2workflow.Sleep(ctx, time.Hour)
3workflow.ExecuteActivity(ctx, ChargeCustomer, order)
4workflow.Go(ctx, func(ctx workflow.Context) {
5    // Deterministic Temporal coroutine
6})

If that feels restrictive, it is. It is also the entire reason state can be reconstructed from an event log instead of a memory dump.

A Temporal application splits into workflows and activities, and the split is not cosmetic.

Workflows hold decisions. Business logic, branching, sequencing:

go
Copy
1if paymentSucceeded {
2    createShipment()
3} else {
4    cancelOrder()
5}

Workflow code decides what should happen. It does not make network calls or write to a database.

Activities do the side effects: charging a card, calling another service, writing to Postgres, uploading an object, sending an email, provisioning a cluster. Workers poll task queues and execute them. The Temporal Service never runs your application code itself.

This is what makes recovery tractable. Decisions are reconstructible from history, and the failure-prone I/O sits behind boundaries you can time out, retry and monitor independently.

Take the same four steps: charge, ship, wait a day, notify. There are several distinct failure windows and they behave differently.

The workflow worker dies after payment completed. The charge result is already in history. A new worker takes the next workflow task, replays, sees payment is done, and moves to shipment creation. The card is not charged again.

The activity worker dies before charging the customer. Temporal eventually times out the attempt and schedules another one according to the retry policy. Activities get a default policy unless you replace it.

The customer was charged, but the worker died before reporting success. This is the one that costs money. Temporal never received a completion event, so from its point of view the activity failed and is eligible for retry. The payment provider has a different opinion, because it already moved the funds.

No engine fixes that for you. Temporal gives you durable orchestration; your downstream systems still owe you safe side-effect semantics. Activities are at-least-once, and the standard answer is a stable idempotency key:

go
Copy
1func ChargeCustomer(ctx context.Context, order Order) error {
2    return paymentProvider.Charge(ctx, ChargeRequest{
3        CustomerID:     order.CustomerID,
4        AmountInCents:  order.AmountInCents,
5        IdempotencyKey: "order-charge:" + order.ID,
6    })
7}

Every retry sends the same key, and the payment service has to return the existing result rather than creating a second charge. Derive the key from workflow data, not from uuid.New(), or you have written a very expensive no-op.

This line looks like time.Sleep and behaves nothing like it.

go
Copy
1err := workflow.Sleep(ctx, 30*24*time.Hour)

time.Sleep needs the process, the goroutine and the memory to stay alive for thirty days. A Temporal timer is an event in the workflow's history. The worker schedules it and can then shut down. Temporal tracks the deadline and creates new work when it fires. On replay, a timer that already fired is not started again.

That makes durable timers the right tool for trial expiry, payment reminders, delayed cancellation, human approval deadlines, infrastructure cleanup, subscription renewals, and retry schedules measured in days. The timer belongs to the workflow's logical execution, not to any one machine.

Retries fix transient technical failures. They do nothing for permanent business failures. If payment succeeds and shipment creation fails forever because the item no longer exists, retrying shipment until the heat death of the universe will not produce a package. You have to undo the payment.

That is a saga: a distributed operation expressed as local transactions, each with a compensating action.

Copy
Charge customer       → Refund customer
Create shipment       → Cancel shipment
Reserve inventory     → Release inventory
Create account        → Disable account

Compensations are business operations, not database snapshots and not ACID rollbacks. Other transactions may have touched the same systems while your workflow was mid-flight, so a compensation has to restore a valid business state rather than replay old data over the top of newer facts. Temporal's own saga guidance is blunt about this: you get eventual consistency, not invisible atomicity.

Here is an abridged version.

go
Copy
1package orders
2
3import (
4    "time"
5
6    "go.temporal.io/sdk/temporal"
7    "go.temporal.io/sdk/workflow"
8)
9
10type Order struct {
11    ID            string
12    CustomerID    string
13    AmountInCents int64
14    Address       string
15}
16
17type compensation struct {
18    name     string
19    activity interface{}
20    args     []interface{}
21}
22
23func OrderWorkflow(ctx workflow.Context, order Order) (err error) {
24    options := workflow.ActivityOptions{
25        StartToCloseTimeout: 30 * time.Second,
26        RetryPolicy: &temporal.RetryPolicy{
27            InitialInterval:    time.Second,
28            BackoffCoefficient: 2,
29            MaximumInterval:    30 * time.Second,
30            MaximumAttempts:    5,
31        },
32    }
33
34    ctx = workflow.WithActivityOptions(ctx, options)
35    logger := workflow.GetLogger(ctx)
36
37    compensations := make([]compensation, 0, 2)
38
39    defer func() {
40        if err == nil {
41            return
42        }
43
44        // The original context might already be cancelled.
45        compensationCtx, _ := workflow.NewDisconnectedContext(ctx)
46
47        // Compensate in reverse order.
48        for i := len(compensations) - 1; i >= 0; i-- {
49            item := compensations[i]
50
51            compensationErr := workflow.ExecuteActivity(
52                compensationCtx,
53                item.activity,
54                item.args...,
55            ).Get(compensationCtx, nil)
56
57            if compensationErr != nil {
58                logger.Error(
59                    "compensation failed",
60                    "name", item.name,
61                    "error", compensationErr,
62                )
63            }
64        }
65    }()
66
67    // Register before charging. RefundIfCharged must safely become
68    // a no-op when no charge was created.
69    compensations = append(compensations, compensation{
70        name:     "refund payment",
71        activity: RefundIfCharged,
72        args:     []interface{}{order},
73    })
74
75    err = workflow.ExecuteActivity(
76        ctx,
77        ChargeCustomer,
78        order,
79    ).Get(ctx, nil)
80    if err != nil {
81        return err
82    }
83
84    // Durable timer: 30 seconds for the demo.
85    // Hours or months in production.
86    if err = workflow.Sleep(ctx, 30*time.Second); err != nil {
87        return err
88    }
89
90    compensations = append(compensations, compensation{
91        name:     "cancel shipment",
92        activity: CancelShipmentIfCreated,
93        args:     []interface{}{order},
94    })
95
96    err = workflow.ExecuteActivity(
97        ctx,
98        CreateShipment,
99        order,
100    ).Get(ctx, nil)
101    if err != nil {
102        return err
103    }
104
105    err = workflow.ExecuteActivity(
106        ctx,
107        NotifyCustomer,
108        order,
109    ).Get(ctx, nil)
110    if err != nil {
111        return err
112    }
113
114    return nil
115}

The detail people get wrong is the ordering: the compensation is registered before the forward activity runs.

Go back to the partial failure from earlier. ChargeCustomer charges the card and loses its connection before returning. The workflow believes the activity failed; the provider believes it succeeded. If you only append the refund after ChargeCustomer returns a nil error, there is no refund registered for exactly the window where you most need one. Registering first closes the gap, which is why the compensating activity has to be idempotent and has to do nothing gracefully when the forward operation never happened. Temporal's official Go samples express saga compensation the same way, directly in workflow code.

Start a local development server:

bash
Copy
1brew install temporal
2temporal server start-dev

The UI comes up on port 8233. Run the worker in another terminal:

bash
Copy
1go run ./cmd/worker

Start an order:

bash
Copy
1go run ./cmd/start-order

Once the payment activity completes and the workflow enters its timer, kill the worker outright:

bash
Copy
1pkill -9 order-worker

The Temporal Service is still up, but no application process is handling that workflow. Bring it back:

bash
Copy
1go run ./cmd/worker

The new worker gets a workflow task, pulls the existing history, and replays. It reconstructs that the customer was charged, the refund compensation is registered, the timer was started, and the workflow is either still waiting on it or has already seen it fire. Then it continues into shipment creation. No restart from zero, no second charge.

Do this before production, not after. Killing every worker and restarting them is the cheapest way to find out whether your replay logic, idempotency, timeouts and retry policies are what you think they are.

Worth repeating, because teams get burned by assuming otherwise. Temporal can guarantee that a workflow observes an activity as completed once. It cannot guarantee the external effect inside that activity happened once.

Copy
1. Worker sends charge request.
2. Payment provider charges customer.
3. Worker crashes.
4. Temporal never receives ActivityTaskCompleted.
5. Activity is retried.

When the acknowledgement is lost, no distributed system can tell whether step two happened. Nothing can. So the production model is:

Copy
Temporal retry
        +
Stable idempotency key
        +
Idempotent downstream operation
        =
Effectively-once business result

The same applies to compensations. RefundIfCharged and CancelShipmentIfCreated have to survive being retried, and have to survive being called when the forward operation never landed.

Temporal deletes a pile of custom infrastructure and hands you constraints in exchange.

Determinism is a versioning problem in disguise. Changing a long-running workflow's control flow means new code can emit commands that disagree with old history. You cannot casually turn this:

go
Copy
1ChargeCustomer()
2CreateShipment()

into this:

go
Copy
1ReserveInventory()
2ChargeCustomer()
3CreateShipment()

while executions started under the old shape are still in flight. Temporal has versioning mechanisms so old histories keep following compatible code while new executions get the new behaviour, and you will need them earlier than you expect.

Activity boundaries need actual design. One giant activity is hard to retry safely. One activity per trivial operation gives you overhead and enormous histories. The useful unit is one meaningful, independently retryable side effect.

Compensation is domain logic, not plumbing. "Refund payment" is easy. "Undo a shipment" may be impossible once a courier has the box. "Unsend an email" is just impossible. Good workflow design classifies failures instead of compensating everything reflexively. A permanently failed notification should probably open a support task, not cancel a shipment that is otherwise fine.

And the Temporal Service becomes critical infrastructure. Your workflows survive worker failure only while the cluster and its persistence layer are healthy, so self-hosting brings capacity planning, database operations, monitoring, backups, upgrades and multi-zone failure planning. Durable execution removes application-level recovery machinery. It does not remove infrastructure engineering, and if you were hoping otherwise, budget for it now.

Temporal starts paying for itself when a process spans multiple services or databases, has to survive deploys and machine failures, needs retries with durable backoff, waits for hours or months, blocks on human approval or an external callback, needs compensation after partial completion, or has to expose its current state so somebody can debug it at 3am. Order fulfilment, subscription lifecycles, provisioning, onboarding, payment flows, data pipelines, training jobs.

For a synchronous CRUD endpoint that does one local transaction and returns in 50ms, it is a very elaborate way to make things worse.

Temporal does not make failures go away. It moves where failure handling lives.

Without a durable engine, an order service grows into a hand-rolled state machine with columns like these:

Copy
payment_status
shipment_status
notification_status
retry_count
next_retry_at
compensation_status
last_error
locked_by
lock_expiry

Spread across database rows, queues, cron jobs, callbacks and recovery scripts, all of it invented locally and none of it tested.

Temporal makes the workflow itself the state machine. The code describes the process, the event history records progress, workers become replaceable compute, activities isolate side effects, retries handle forward recovery, compensations handle backward recovery, and durable timers let execution wait without holding a process hostage.

You are still building a distributed state machine. You are just writing it as ordinary application code and letting the engine remember where the system is once every worker has forgotten.