Building reliable distributed systems is challenging. Network failures, service outages, and unexpected errors can leave your workflows in inconsistent states. What if there was a way to build workflows that are inherently resilient, automatically handling retries, timeouts, and state management?

Enter Temporal.io - a workflow orchestration platform that makes building reliable distributed applications dramatically easier. In this comprehensive tutorial, we’ll build a coffee shop ordering system that demonstrates Temporal’s powerful capabilities.

What is Temporal.io?

Temporal is a microservice orchestration platform that simplifies the development of reliable distributed applications. It provides:

  • Durable Execution: Your workflow code runs to completion, even if it takes days or weeks
  • Automatic Retries: Activities automatically retry on failure with configurable policies
  • State Management: Workflow state is automatically persisted and recovered
  • Visibility: Built-in observability into workflow execution history
  • Time Travel: Replay workflows to debug issues or understand behavior

Think of Temporal as a “database for workflow state” - it ensures your business logic executes reliably, even in the face of failures.

The Coffee Shop Scenario

Let’s build a realistic coffee shop ordering system with the following workflow:

  1. Receive Order: Customer places an order for their favorite coffee
  2. Process Payment: Charge the customer’s payment method
  3. Prepare Beverage: Barista prepares the drink
  4. Update Inventory: Deduct ingredients from inventory
  5. Notify Customer: Send notification when order is ready

Each step can fail, and we need to handle failures gracefully with compensating actions (e.g., refund payment if beverage preparation fails).

System Architecture

graph TB subgraph "Client Layer" Client[Coffee Shop App] end subgraph "Temporal Cluster" TemporalServer[Temporal Server] WorkflowEngine[Workflow Engine] History[Event History Store] end subgraph "Worker Services" Worker[Temporal Worker] Workflow[Order Workflow] Activities[Activities] end subgraph "External Services" Payment[Payment Service] Inventory[Inventory Service] Notification[Notification Service] end Client -->|Start Workflow| TemporalServer TemporalServer --> WorkflowEngine WorkflowEngine <--> History Worker -->|Poll for Tasks| TemporalServer Worker --> Workflow Workflow --> Activities Activities --> Payment Activities --> Inventory Activities --> Notification style Workflow fill:#e1f5ff style TemporalServer fill:#fff4e1 style Activities fill:#e8f5e9

Setting Up the Project

First, let’s set up our Go project with Temporal dependencies:

// go.mod
module coffee-shop

go 1.21

require (
    go.temporal.io/sdk v1.25.1
    github.com/google/uuid v1.5.0
)

Install dependencies:

go mod download

You’ll also need to run a local Temporal server. The easiest way is using Docker:

docker run -d -p 7233:7233 -p 8233:8233 temporalio/auto-setup:latest

Defining Our Domain Models

// models/order.go
package models

import "time"

type Order struct {
    OrderID     string
    CustomerID  string
    Items       []OrderItem
    TotalAmount float64
    Status      OrderStatus
    CreatedAt   time.Time
}

type OrderItem struct {
    ItemName string
    Quantity int
    Price    float64
}

type OrderStatus string

const (
    OrderStatusPending    OrderStatus = "PENDING"
    OrderStatusPaid       OrderStatus = "PAID"
    OrderStatusPreparing  OrderStatus = "PREPARING"
    OrderStatusReady      OrderStatus = "READY"
    OrderStatusCompleted  OrderStatus = "COMPLETED"
    OrderStatusFailed     OrderStatus = "FAILED"
    OrderStatusRefunded   OrderStatus = "REFUNDED"
)

type PaymentInfo struct {
    PaymentMethod string
    Amount        float64
    TransactionID string
}

Implementing Activities

Activities are the building blocks of Temporal workflows - they represent individual tasks that interact with external systems.

// activities/coffee_activities.go
package activities

import (
    "context"
    "fmt"
    "time"

    "coffee-shop/models"
    "github.com/google/uuid"
    "go.temporal.io/sdk/activity"
)

type CoffeeShopActivities struct{}

// ProcessPayment charges the customer's payment method
func (a *CoffeeShopActivities) ProcessPayment(ctx context.Context, order models.Order) (*models.PaymentInfo, error) {
    logger := activity.GetLogger(ctx)
    logger.Info("Processing payment", "orderID", order.OrderID, "amount", order.TotalAmount)

    // Simulate payment processing
    time.Sleep(2 * time.Second)

    // Simulate occasional payment failures (10% chance)
    // if rand.Float32() < 0.1 {
    //     return nil, fmt.Errorf("payment declined")
    // }

    payment := &models.PaymentInfo{
        PaymentMethod: "credit_card",
        Amount:        order.TotalAmount,
        TransactionID: uuid.New().String(),
    }

    logger.Info("Payment processed successfully", "transactionID", payment.TransactionID)
    return payment, nil
}

// RefundPayment reverses a payment (compensating action)
func (a *CoffeeShopActivities) RefundPayment(ctx context.Context, payment models.PaymentInfo) error {
    logger := activity.GetLogger(ctx)
    logger.Info("Refunding payment", "transactionID", payment.TransactionID)

    time.Sleep(1 * time.Second)

    logger.Info("Payment refunded successfully")
    return nil
}

// PrepareBeverage simulates the barista making the drink
func (a *CoffeeShopActivities) PrepareBeverage(ctx context.Context, order models.Order) error {
    logger := activity.GetLogger(ctx)
    logger.Info("Preparing beverage", "orderID", order.OrderID)

    // Simulate beverage preparation time
    time.Sleep(5 * time.Second)

    for _, item := range order.Items {
        logger.Info("Preparing item", "name", item.ItemName, "quantity", item.Quantity)
    }

    // Simulate occasional preparation failures (5% chance)
    // if rand.Float32() < 0.05 {
    //     return fmt.Errorf("out of ingredients")
    // }

    logger.Info("Beverage prepared successfully")
    return nil
}

// UpdateInventory deducts ingredients from inventory
func (a *CoffeeShopActivities) UpdateInventory(ctx context.Context, order models.Order) error {
    logger := activity.GetLogger(ctx)
    logger.Info("Updating inventory", "orderID", order.OrderID)

    time.Sleep(1 * time.Second)

    for _, item := range order.Items {
        logger.Info("Deducting inventory", "item", item.ItemName, "quantity", item.Quantity)
    }

    return nil
}

// RestoreInventory adds ingredients back to inventory (compensating action)
func (a *CoffeeShopActivities) RestoreInventory(ctx context.Context, order models.Order) error {
    logger := activity.GetLogger(ctx)
    logger.Info("Restoring inventory", "orderID", order.OrderID)

    time.Sleep(1 * time.Second)

    for _, item := range order.Items {
        logger.Info("Restoring inventory", "item", item.ItemName, "quantity", item.Quantity)
    }

    return nil
}

// NotifyCustomer sends a notification to the customer
func (a *CoffeeShopActivities) NotifyCustomer(ctx context.Context, orderID string, message string) error {
    logger := activity.GetLogger(ctx)
    logger.Info("Sending notification", "orderID", orderID, "message", message)

    time.Sleep(500 * time.Millisecond)

    // In a real system, this would send an email, SMS, or push notification
    fmt.Printf("📱 Notification sent for order %s: %s\n", orderID, message)

    return nil
}

The Workflow Implementation

The workflow orchestrates our activities and handles failures with compensations.

// workflows/order_workflow.go
package workflows

import (
    "fmt"
    "time"

    "coffee-shop/activities"
    "coffee-shop/models"
    "go.temporal.io/sdk/workflow"
)

// OrderWorkflow orchestrates the entire coffee order process
func OrderWorkflow(ctx workflow.Context, order models.Order) error {
    logger := workflow.GetLogger(ctx)
    logger.Info("Starting order workflow", "orderID", order.OrderID)

    // Configure activity options with retry policy
    activityOptions := workflow.ActivityOptions{
        StartToCloseTimeout: 30 * time.Second,
        RetryPolicy: &temporal.RetryPolicy{
            InitialInterval:    time.Second,
            BackoffCoefficient: 2.0,
            MaximumInterval:    100 * time.Second,
            MaximumAttempts:    3,
        },
    }
    ctx = workflow.WithActivityOptions(ctx, activityOptions)

    var a *activities.CoffeeShopActivities
    var payment *models.PaymentInfo
    var paymentProcessed bool
    var inventoryUpdated bool

    // Step 1: Process Payment
    logger.Info("Step 1: Processing payment")
    err := workflow.ExecuteActivity(ctx, a.ProcessPayment, order).Get(ctx, &payment)
    if err != nil {
        logger.Error("Payment processing failed", "error", err)
        _ = workflow.ExecuteActivity(ctx, a.NotifyCustomer, order.OrderID,
            "Your order failed: Payment could not be processed").Get(ctx, nil)
        return fmt.Errorf("payment failed: %w", err)
    }
    paymentProcessed = true
    logger.Info("Payment processed", "transactionID", payment.TransactionID)

    // Step 2: Prepare Beverage
    logger.Info("Step 2: Preparing beverage")
    err = workflow.ExecuteActivity(ctx, a.PrepareBeverage, order).Get(ctx, nil)
    if err != nil {
        logger.Error("Beverage preparation failed", "error", err)

        // Compensate: Refund payment
        if paymentProcessed {
            logger.Info("Compensating: Refunding payment")
            _ = workflow.ExecuteActivity(ctx, a.RefundPayment, *payment).Get(ctx, nil)
        }

        _ = workflow.ExecuteActivity(ctx, a.NotifyCustomer, order.OrderID,
            "Your order failed: Unable to prepare beverage. Payment has been refunded.").Get(ctx, nil)
        return fmt.Errorf("beverage preparation failed: %w", err)
    }
    logger.Info("Beverage prepared successfully")

    // Step 3: Update Inventory
    logger.Info("Step 3: Updating inventory")
    err = workflow.ExecuteActivity(ctx, a.UpdateInventory, order).Get(ctx, nil)
    if err != nil {
        logger.Error("Inventory update failed", "error", err)

        // Compensate: Refund payment (we can't "unprepare" the beverage)
        if paymentProcessed {
            logger.Info("Compensating: Refunding payment")
            _ = workflow.ExecuteActivity(ctx, a.RefundPayment, *payment).Get(ctx, nil)
        }

        _ = workflow.ExecuteActivity(ctx, a.NotifyCustomer, order.OrderID,
            "Your order failed: Inventory error. Payment has been refunded.").Get(ctx, nil)
        return fmt.Errorf("inventory update failed: %w", err)
    }
    inventoryUpdated = true
    logger.Info("Inventory updated successfully")

    // Step 4: Notify Customer
    logger.Info("Step 4: Notifying customer")
    err = workflow.ExecuteActivity(ctx, a.NotifyCustomer, order.OrderID,
        "Your order is ready for pickup!").Get(ctx, nil)
    if err != nil {
        logger.Warn("Customer notification failed, but order is complete", "error", err)
        // We don't fail the workflow if notification fails
    }

    logger.Info("Order workflow completed successfully", "orderID", order.OrderID)
    return nil
}

Workflow State Diagram

stateDiagram-v2 [*] --> PENDING: Order Created PENDING --> ProcessPayment: Start Workflow ProcessPayment --> PAID: Payment Success ProcessPayment --> FAILED: Payment Failed PAID --> PrepareBeverage: Payment Confirmed PrepareBeverage --> PREPARING: Preparation Started PREPARING --> UpdateInventory: Beverage Ready PREPARING --> RefundPayment: Preparation Failed UpdateInventory --> READY: Inventory Updated UpdateInventory --> RefundPayment: Inventory Error READY --> NotifyCustomer: Order Complete NotifyCustomer --> COMPLETED: Notification Sent NotifyCustomer --> COMPLETED: Notification Failed (Non-Critical) RefundPayment --> REFUNDED: Refund Complete FAILED --> [*]: Workflow Failed REFUNDED --> [*]: Workflow Compensated COMPLETED --> [*]: Workflow Success note right of ProcessPayment Retries automatically on transient failures end note note right of RefundPayment Compensating action reverses payment end note

Activity Execution Flow

sequenceDiagram participant C as Client participant T as Temporal Server participant W as Worker participant P as Payment Service participant B as Barista participant I as Inventory participant N as Notification Service C->>T: StartWorkflow(order) T->>W: Schedule Workflow activate W W->>T: Schedule ProcessPayment Activity T->>W: Execute Activity W->>P: Charge Payment P-->>W: Payment Success W->>T: Activity Complete W->>T: Schedule PrepareBeverage Activity T->>W: Execute Activity W->>B: Prepare Order B-->>W: Beverage Ready W->>T: Activity Complete W->>T: Schedule UpdateInventory Activity T->>W: Execute Activity W->>I: Deduct Ingredients I-->>W: Inventory Updated W->>T: Activity Complete W->>T: Schedule NotifyCustomer Activity T->>W: Execute Activity W->>N: Send Notification N-->>W: Notification Sent W->>T: Activity Complete W->>T: Workflow Complete deactivate W T-->>C: Workflow Result

Worker Setup

The worker is responsible for executing workflows and activities:

// worker/main.go
package main

import (
    "log"

    "coffee-shop/activities"
    "coffee-shop/workflows"
    "go.temporal.io/sdk/client"
    "go.temporal.io/sdk/worker"
)

func main() {
    // Create Temporal client
    c, err := client.Dial(client.Options{
        HostPort: "localhost:7233",
    })
    if err != nil {
        log.Fatalln("Unable to create Temporal client", err)
    }
    defer c.Close()

    // Create worker
    w := worker.New(c, "coffee-shop-task-queue", worker.Options{})

    // Register workflows
    w.RegisterWorkflow(workflows.OrderWorkflow)

    // Register activities
    coffeeActivities := &activities.CoffeeShopActivities{}
    w.RegisterActivity(coffeeActivities)

    // Start worker
    log.Println("Starting Coffee Shop worker...")
    err = w.Run(worker.InterruptCh())
    if err != nil {
        log.Fatalln("Unable to start worker", err)
    }
}

Client Application

Finally, let’s create a client to start workflows:

// cmd/client/main.go
package main

import (
    "context"
    "fmt"
    "log"
    "time"

    "coffee-shop/models"
    "coffee-shop/workflows"
    "github.com/google/uuid"
    "go.temporal.io/sdk/client"
)

func main() {
    // Create Temporal client
    c, err := client.Dial(client.Options{
        HostPort: "localhost:7233",
    })
    if err != nil {
        log.Fatalln("Unable to create Temporal client", err)
    }
    defer c.Close()

    // Create an order
    order := models.Order{
        OrderID:    uuid.New().String(),
        CustomerID: "customer-123",
        Items: []models.OrderItem{
            {ItemName: "Cappuccino", Quantity: 1, Price: 4.50},
            {ItemName: "Croissant", Quantity: 1, Price: 3.00},
        },
        TotalAmount: 7.50,
        Status:      models.OrderStatusPending,
        CreatedAt:   time.Now(),
    }

    fmt.Printf("☕ Placing order: %s\n", order.OrderID)

    // Start workflow
    workflowOptions := client.StartWorkflowOptions{
        ID:        "coffee-order-" + order.OrderID,
        TaskQueue: "coffee-shop-task-queue",
    }

    we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, workflows.OrderWorkflow, order)
    if err != nil {
        log.Fatalln("Unable to execute workflow", err)
    }

    fmt.Printf("Started workflow - WorkflowID: %s, RunID: %s\n", we.GetID(), we.GetRunID())

    // Wait for workflow to complete
    var result interface{}
    err = we.Get(context.Background(), &result)
    if err != nil {
        log.Fatalln("Workflow failed", err)
    }

    fmt.Println("✅ Order completed successfully!")
}

Running the Application

  1. Start Temporal Server (if not already running):
docker run -d -p 7233:7233 -p 8233:8233 temporalio/auto-setup:latest
  1. Start the Worker:
go run worker/main.go
  1. Place an Order (in a new terminal):
go run cmd/client/main.go
  1. View Workflow in Temporal UI: Open http://localhost:8233 in your browser to see the workflow execution history.

Failure Scenarios and Compensation

One of Temporal’s most powerful features is its ability to handle failures gracefully. Let’s visualize the compensation flow:

flowchart TD Start([Order Placed]) --> Payment[Process Payment] Payment -->|Success| PaymentOK{Payment OK} Payment -->|Failure| NotifyFail1[Notify: Payment Failed] NotifyFail1 --> End1([End: Failed]) PaymentOK --> Prepare[Prepare Beverage] Prepare -->|Success| PrepareOK{Beverage Ready} Prepare -->|Failure| Compensate1[Refund Payment] Compensate1 --> NotifyFail2[Notify: Refunded] NotifyFail2 --> End2([End: Compensated]) PrepareOK --> Inventory[Update Inventory] Inventory -->|Success| InventoryOK{Inventory Updated} Inventory -->|Failure| Compensate2[Refund Payment] Compensate2 --> NotifyFail3[Notify: Refunded] NotifyFail3 --> End3([End: Compensated]) InventoryOK --> Notify[Notify Customer] Notify --> Complete([End: Success]) style Compensate1 fill:#ffcccc style Compensate2 fill:#ffcccc style Complete fill:#ccffcc style End1 fill:#ffcccc style End2 fill:#ffffcc style End3 fill:#ffffcc

Advanced Features

1. Timeouts and Heartbeats

For long-running activities, use heartbeats to detect worker crashes:

func (a *CoffeeShopActivities) PrepareBeverage(ctx context.Context, order models.Order) error {
    logger := activity.GetLogger(ctx)

    for i := 0; i < 5; i++ {
        // Record heartbeat to let Temporal know we're still alive
        activity.RecordHeartbeat(ctx, i)

        logger.Info("Preparing step", "step", i+1)
        time.Sleep(1 * time.Second)
    }

    return nil
}

2. Signals for Dynamic Updates

Allow external events to modify running workflows:

func OrderWorkflow(ctx workflow.Context, order models.Order) error {
    // ... existing code ...

    // Create signal channel for order cancellation
    cancelChannel := workflow.GetSignalChannel(ctx, "cancel-order")

    selector := workflow.NewSelector(ctx)

    // Listen for cancellation signal
    selector.AddReceive(cancelChannel, func(c workflow.ReceiveChannel, more bool) {
        var cancelReason string
        c.Receive(ctx, &cancelReason)

        logger.Info("Order cancelled", "reason", cancelReason)
        // Trigger compensations
    })

    // ... rest of workflow ...
}

3. Queries for Workflow State

Query running workflows without affecting their execution:

func OrderWorkflow(ctx workflow.Context, order models.Order) error {
    currentStatus := models.OrderStatusPending

    // Register query handler
    err := workflow.SetQueryHandler(ctx, "status", func() (models.OrderStatus, error) {
        return currentStatus, nil
    })
    if err != nil {
        return err
    }

    // Update status as workflow progresses
    currentStatus = models.OrderStatusPaid
    // ... continue workflow ...
}

Query from client:

response, err := c.QueryWorkflow(ctx, workflowID, runID, "status")
var status models.OrderStatus
response.Get(&status)
fmt.Printf("Current order status: %s\n", status)

Key Concepts Recap

Workflows:

  • Orchestrate business logic
  • Must be deterministic (no random numbers, no direct I/O)
  • Automatically retried and replayed
  • Can run for days, weeks, or months

Activities:

  • Perform actual work (API calls, database operations, etc.)
  • Can be non-deterministic
  • Automatically retried with configurable policies
  • Should be idempotent when possible

Task Queue:

  • Named queue that connects workflows to workers
  • Workers poll task queues for work
  • Provides load balancing and scalability

Workers:

  • Execute workflow and activity code
  • Can be scaled horizontally
  • Poll Temporal Server for tasks

Best Practices

  1. Keep Workflows Deterministic: Don’t use time.Now(), rand, or I/O in workflows. Use Temporal’s workflow.Now() and workflow.Sleep() instead.

  2. Make Activities Idempotent: Activities may be retried, so design them to safely execute multiple times.

  3. Use Appropriate Timeouts: Configure StartToCloseTimeout for activities to prevent hanging workflows.

  4. Implement Compensations: For saga patterns, implement compensating activities to undo work.

  5. Use Signals for External Events: Allow external systems to interact with running workflows via signals.

  6. Monitor with Temporal UI: Use the web UI to understand workflow execution and debug issues.

  7. Version Workflows Carefully: When changing workflows, use versioning APIs to handle in-flight workflows.

Complete Workflow Lifecycle

graph LR subgraph "Client" A[Start Workflow] end subgraph "Temporal Server" B[Create Workflow Instance] C[Store in Event History] D[Schedule Tasks] end subgraph "Worker Pool" E[Worker 1] F[Worker 2] G[Worker N] end subgraph "External Services" H[Payment API] I[Inventory DB] J[Notification Service] end A --> B B --> C C --> D D -.Poll for Tasks.-> E D -.Poll for Tasks.-> F D -.Poll for Tasks.-> G E --> H F --> I G --> J H -.Result.-> E I -.Result.-> F J -.Result.-> G E --> C F --> C G --> C style A fill:#e3f2fd style B fill:#fff3e0 style C fill:#f3e5f5 style D fill:#e8f5e9

Conclusion

Temporal.io transforms how we build distributed applications. Instead of manually managing state, retries, and failure scenarios, Temporal provides a robust framework that handles these complexities automatically.

Our coffee shop example demonstrates:

  • Resilient workflows that survive failures and restarts
  • Automatic retries with configurable policies
  • Compensation logic for saga patterns
  • Clear separation between orchestration (workflows) and execution (activities)
  • Built-in observability through event history

Whether you’re building order processing systems, data pipelines, or complex microservice orchestrations, Temporal provides the foundation for reliable, maintainable distributed applications.

Next Steps

Happy coding, and may your workflows always complete successfully! ☕

References