Microservices

Building reliable distributed systems is challenging. Network failures, service outages, and unexpected errors can leave your workflows in inconsistent states. What if there was a way to build workflows that are inherently resilient, automatically handling retries, timeouts, and state management? Enter Temporal.io - a workflow orchestration platform that makes building reliable distributed applications dramatically easier. In this comprehensive tutorial, we’ll build a coffee shop ordering system that demonstrates Temporal’s powerful capabilities. ...

The future of software development isn’t just about AI-it’s about AI agents: autonomous systems that can reason, plan, and execute complex tasks with minimal human intervention. And as we stand on the precipice of this transformation, one programming language is uniquely positioned to dominate the agent era: Go. In this deep dive, we’ll explore why AI agents represent the next evolutionary leap in software, examine the technical requirements for building robust agent systems, and demonstrate why Go’s design philosophy makes it the ideal foundation for this new paradigm. ...

In the world of modern software development, the question isn’t whether you’ll need to scale-it’s when. If you’ve ever watched a monolithic application groan under increasing load, fought to deploy a single feature without breaking everything else, or felt trapped by technology choices made years ago, you’re not alone. Let’s explore how event-driven microservices in Go can solve these challenges and build systems that scale gracefully with your ambitions. The Pain of the Monolith Picture this: Your application has grown from a simple CRUD app to a complex beast handling users, notes, notifications, analytics, and more. Every deployment is a nail-biting experience because changing one module might break three others. Your database has become a bottleneck, and adding more servers doesn’t help because everything shares the same database connection pool. Different teams step on each other’s toes, and that cool new technology? Sorry, the entire stack is locked into decisions made in 2015. ...

Go Architecture Patterns Series: ← Previous: Event Sourcing | Series Overview What is the Saga Pattern? The Saga pattern is a design pattern for managing distributed transactions across multiple microservices. Instead of using traditional ACID transactions, a saga breaks a transaction into a sequence of local transactions, each with a compensating transaction to undo changes if needed. Key Principles: Local Transactions: Each service executes its own local transaction Compensating Transactions: Undo operations for rollback Eventual Consistency: System reaches consistent state over time Choreography or Orchestration: Two approaches to coordinate sagas Idempotency: Operations can be safely retried Error Handling: Graceful handling of failures Architecture Overview %%{init: {'theme':'dark'}}%% graph TB subgraph "Orchestration-Based Saga" Orchestrator[Saga Orchestrator] S1[Service 1] S2[Service 2] S3[Service 3] S4[Service 4] Orchestrator -->|1. Execute| S1 Orchestrator -->|2. Execute| S2 Orchestrator -->|3. Execute| S3 Orchestrator -->|4. Execute| S4 S1 -.->|Failed| Orchestrator S2 -.->|Success| Orchestrator S3 -.->|Failed| Orchestrator Orchestrator -.->|Compensate| S2 Orchestrator -.->|Compensate| S1 end subgraph "Choreography-Based Saga" EventBus[Event Bus] OrderSvc[Order Service] PaymentSvc[Payment Service] InventorySvc[Inventory Service] ShippingSvc[Shipping Service] OrderSvc -->|OrderCreated| EventBus EventBus -->|OrderCreated| PaymentSvc PaymentSvc -->|PaymentProcessed| EventBus EventBus -->|PaymentProcessed| InventorySvc InventorySvc -->|InventoryReserved| EventBus EventBus -->|InventoryReserved| ShippingSvc PaymentSvc -.->|PaymentFailed| EventBus EventBus -.->|PaymentFailed| OrderSvc end Saga Execution Flow %%{init: {'theme':'dark', 'themeVariables': {'primaryTextColor':'#e5e7eb','secondaryTextColor':'#e5e7eb','tertiaryTextColor':'#e5e7eb','textColor':'#e5e7eb','nodeTextColor':'#e5e7eb','edgeLabelText':'#e5e7eb','clusterTextColor':'#e5e7eb','actorTextColor':'#e5e7eb'}}}%% sequenceDiagram participant Client participant Orchestrator participant OrderSvc participant PaymentSvc participant InventorySvc participant ShippingSvc Note over Client,ShippingSvc: Happy Path Client->>Orchestrator: Create Order Orchestrator->>OrderSvc: Create Order OrderSvc-->>Orchestrator: Order Created Orchestrator->>PaymentSvc: Process Payment PaymentSvc-->>Orchestrator: Payment Success Orchestrator->>InventorySvc: Reserve Inventory InventorySvc-->>Orchestrator: Inventory Reserved Orchestrator->>ShippingSvc: Schedule Shipping ShippingSvc-->>Orchestrator: Shipping Scheduled Orchestrator-->>Client: Order Completed Note over Client,ShippingSvc: Failure with Compensation Client->>Orchestrator: Create Order Orchestrator->>OrderSvc: Create Order OrderSvc-->>Orchestrator: Order Created Orchestrator->>PaymentSvc: Process Payment PaymentSvc-->>Orchestrator: Payment Success Orchestrator->>InventorySvc: Reserve Inventory InventorySvc-->>Orchestrator: Insufficient Stock Note over Orchestrator: Trigger Compensation Orchestrator->>PaymentSvc: Refund Payment PaymentSvc-->>Orchestrator: Payment Refunded Orchestrator->>OrderSvc: Cancel Order OrderSvc-->>Orchestrator: Order Cancelled Orchestrator-->>Client: Order Failed (Compensated) Real-World Use Cases E-commerce Order Processing: Order, payment, inventory, shipping coordination Travel Booking Systems: Flight, hotel, car rental bookings Financial Transactions: Multi-step payment processing Supply Chain Management: Order fulfillment across warehouses Banking Systems: Money transfers across accounts Healthcare Systems: Patient admission, bed assignment, billing Saga Pattern Implementation Project Structure saga-app/ ├── cmd/ │ └── api/ │ └── main.go ├── internal/ │ ├── saga/ │ │ ├── orchestrator.go │ │ ├── step.go │ │ └── state.go │ ├── services/ │ │ ├── order/ │ │ ├── payment/ │ │ ├── inventory/ │ │ └── shipping/ │ └── models/ │ └── order.go └── go.mod Saga Step Definition // internal/saga/step.go package saga import ( "context" "fmt" ) // StepStatus represents the status of a saga step type StepStatus string const ( StepStatusPending StepStatus = "pending" StepStatusInProgress StepStatus = "in_progress" StepStatusCompleted StepStatus = "completed" StepStatusFailed StepStatus = "failed" StepStatusCompensated StepStatus = "compensated" ) // Step represents a single step in a saga type Step struct { Name string Execute ExecuteFunc Compensate CompensateFunc Status StepStatus RetryCount int MaxRetries int CompensationData interface{} } // ExecuteFunc is the function that executes a step type ExecuteFunc func(ctx context.Context, data interface{}) (interface{}, error) // CompensateFunc is the function that compensates a step type CompensateFunc func(ctx context.Context, data interface{}) error // NewStep creates a new saga step func NewStep(name string, execute ExecuteFunc, compensate CompensateFunc) *Step { return &Step{ Name: name, Execute: execute, Compensate: compensate, Status: StepStatusPending, MaxRetries: 3, } } // Run executes the step func (s *Step) Run(ctx context.Context, data interface{}) (interface{}, error) { s.Status = StepStatusInProgress result, err := s.Execute(ctx, data) if err != nil { s.Status = StepStatusFailed return nil, fmt.Errorf("step %s failed: %w", s.Name, err) } s.Status = StepStatusCompleted s.CompensationData = result return result, nil } // Compensate executes the compensation func (s *Step) Compensate(ctx context.Context) error { if s.Status != StepStatusCompleted { return nil // Only compensate completed steps } if err := s.Compensate(ctx, s.CompensationData); err != nil { return fmt.Errorf("compensation for %s failed: %w", s.Name, err) } s.Status = StepStatusCompensated return nil } Saga Orchestrator // internal/saga/orchestrator.go package saga import ( "context" "fmt" "log" ) // SagaStatus represents the status of a saga type SagaStatus string const ( SagaStatusPending SagaStatus = "pending" SagaStatusInProgress SagaStatus = "in_progress" SagaStatusCompleted SagaStatus = "completed" SagaStatusFailed SagaStatus = "failed" SagaStatusCompensated SagaStatus = "compensated" ) // Saga represents a distributed transaction type Saga struct { ID string Steps []*Step Status SagaStatus Data interface{} } // Orchestrator manages saga execution type Orchestrator struct { sagas map[string]*Saga } // NewOrchestrator creates a new saga orchestrator func NewOrchestrator() *Orchestrator { return &Orchestrator{ sagas: make(map[string]*Saga), } } // NewSaga creates a new saga func (o *Orchestrator) NewSaga(id string, steps []*Step) *Saga { saga := &Saga{ ID: id, Steps: steps, Status: SagaStatusPending, } o.sagas[id] = saga return saga } // Execute runs the saga func (o *Orchestrator) Execute(ctx context.Context, saga *Saga, initialData interface{}) error { saga.Status = SagaStatusInProgress saga.Data = initialData completedSteps := make([]*Step, 0) var currentData interface{} = initialData // Execute steps sequentially for _, step := range saga.Steps { log.Printf("Executing step: %s", step.Name) result, err := step.Run(ctx, currentData) if err != nil { log.Printf("Step %s failed: %v", step.Name, err) saga.Status = SagaStatusFailed // Trigger compensation if compErr := o.compensate(ctx, completedSteps); compErr != nil { log.Printf("Compensation failed: %v", compErr) return fmt.Errorf("saga failed and compensation failed: %w", compErr) } saga.Status = SagaStatusCompensated return fmt.Errorf("saga failed at step %s: %w", step.Name, err) } completedSteps = append(completedSteps, step) currentData = result log.Printf("Step %s completed successfully", step.Name) } saga.Status = SagaStatusCompleted log.Printf("Saga %s completed successfully", saga.ID) return nil } // compensate executes compensation for completed steps in reverse order func (o *Orchestrator) compensate(ctx context.Context, steps []*Step) error { log.Println("Starting compensation...") // Compensate in reverse order for i := len(steps) - 1; i >= 0; i-- { step := steps[i] log.Printf("Compensating step: %s", step.Name) if err := step.Compensate(ctx); err != nil { return fmt.Errorf("compensation failed for step %s: %w", step.Name, err) } log.Printf("Step %s compensated successfully", step.Name) } log.Println("Compensation completed") return nil } // GetSaga retrieves a saga by ID func (o *Orchestrator) GetSaga(id string) (*Saga, error) { saga, exists := o.sagas[id] if !exists { return nil, fmt.Errorf("saga not found: %s", id) } return saga, nil } Service Implementations // internal/services/order/service.go package order import ( "context" "fmt" "time" ) type Order struct { ID string UserID string Items []OrderItem Total float64 Status string CreatedAt time.Time } type OrderItem struct { ProductID string Quantity int Price float64 } type Service struct { orders map[string]*Order } func NewService() *Service { return &Service{ orders: make(map[string]*Order), } } func (s *Service) CreateOrder(ctx context.Context, userID string, items []OrderItem) (*Order, error) { var total float64 for _, item := range items { total += item.Price * float64(item.Quantity) } order := &Order{ ID: generateID(), UserID: userID, Items: items, Total: total, Status: "pending", CreatedAt: time.Now(), } s.orders[order.ID] = order fmt.Printf("Order created: %s\n", order.ID) return order, nil } func (s *Service) CancelOrder(ctx context.Context, orderID string) error { order, exists := s.orders[orderID] if !exists { return fmt.Errorf("order not found") } order.Status = "cancelled" fmt.Printf("Order cancelled: %s\n", orderID) return nil } func generateID() string { return fmt.Sprintf("ord_%d", time.Now().UnixNano()) } // internal/services/payment/service.go package payment import ( "context" "fmt" "time" ) type Payment struct { ID string OrderID string Amount float64 Status string TransactionID string CreatedAt time.Time } type Service struct { payments map[string]*Payment } func NewService() *Service { return &Service{ payments: make(map[string]*Payment), } } func (s *Service) ProcessPayment(ctx context.Context, orderID string, amount float64) (*Payment, error) { // Simulate payment processing if amount > 10000 { return nil, fmt.Errorf("amount exceeds limit") } payment := &Payment{ ID: generateID(), OrderID: orderID, Amount: amount, Status: "completed", TransactionID: fmt.Sprintf("txn_%d", time.Now().UnixNano()), CreatedAt: time.Now(), } s.payments[payment.ID] = payment fmt.Printf("Payment processed: %s for order %s\n", payment.ID, orderID) return payment, nil } func (s *Service) RefundPayment(ctx context.Context, paymentID string) error { payment, exists := s.payments[paymentID] if !exists { return fmt.Errorf("payment not found") } payment.Status = "refunded" fmt.Printf("Payment refunded: %s\n", paymentID) return nil } func generateID() string { return fmt.Sprintf("pay_%d", time.Now().UnixNano()) } // internal/services/inventory/service.go package inventory import ( "context" "fmt" ) type Reservation struct { ID string ProductID string Quantity int OrderID string } type Service struct { stock map[string]int reservations map[string]*Reservation } func NewService() *Service { return &Service{ stock: map[string]int{ "product_1": 100, "product_2": 50, "product_3": 200, }, reservations: make(map[string]*Reservation), } } func (s *Service) ReserveInventory(ctx context.Context, orderID, productID string, quantity int) (*Reservation, error) { currentStock, exists := s.stock[productID] if !exists { return nil, fmt.Errorf("product not found") } if currentStock < quantity { return nil, fmt.Errorf("insufficient stock: have %d, need %d", currentStock, quantity) } s.stock[productID] = currentStock - quantity reservation := &Reservation{ ID: generateID(), ProductID: productID, Quantity: quantity, OrderID: orderID, } s.reservations[reservation.ID] = reservation fmt.Printf("Inventory reserved: %d units of %s for order %s\n", quantity, productID, orderID) return reservation, nil } func (s *Service) ReleaseReservation(ctx context.Context, reservationID string) error { reservation, exists := s.reservations[reservationID] if !exists { return fmt.Errorf("reservation not found") } s.stock[reservation.ProductID] += reservation.Quantity delete(s.reservations, reservationID) fmt.Printf("Reservation released: %s\n", reservationID) return nil } func generateID() string { return fmt.Sprintf("res_%d", time.Now().UnixNano()) } // internal/services/shipping/service.go package shipping import ( "context" "fmt" "time" ) type Shipment struct { ID string OrderID string TrackingNumber string Status string ScheduledDate time.Time } type Service struct { shipments map[string]*Shipment } func NewService() *Service { return &Service{ shipments: make(map[string]*Shipment), } } func (s *Service) ScheduleShipment(ctx context.Context, orderID string) (*Shipment, error) { shipment := &Shipment{ ID: generateID(), OrderID: orderID, TrackingNumber: fmt.Sprintf("TRK%d", time.Now().UnixNano()), Status: "scheduled", ScheduledDate: time.Now().Add(24 * time.Hour), } s.shipments[shipment.ID] = shipment fmt.Printf("Shipment scheduled: %s for order %s\n", shipment.ID, orderID) return shipment, nil } func (s *Service) CancelShipment(ctx context.Context, shipmentID string) error { shipment, exists := s.shipments[shipmentID] if !exists { return fmt.Errorf("shipment not found") } shipment.Status = "cancelled" fmt.Printf("Shipment cancelled: %s\n", shipmentID) return nil } func generateID() string { return fmt.Sprintf("shp_%d", time.Now().UnixNano()) } Saga Workflow Implementation // internal/saga/order_saga.go package saga import ( "context" "fmt" "app/internal/services/inventory" "app/internal/services/order" "app/internal/services/payment" "app/internal/services/shipping" ) type OrderSagaData struct { UserID string Items []order.OrderItem OrderID string PaymentID string ReservationID string ShipmentID string } // CreateOrderSaga creates a saga for order processing func CreateOrderSaga( orderSvc *order.Service, paymentSvc *payment.Service, inventorySvc *inventory.Service, shippingSvc *shipping.Service, ) []*Step { // Step 1: Create Order createOrderStep := NewStep( "CreateOrder", func(ctx context.Context, data interface{}) (interface{}, error) { sagaData := data.(*OrderSagaData) order, err := orderSvc.CreateOrder(ctx, sagaData.UserID, sagaData.Items) if err != nil { return nil, err } sagaData.OrderID = order.ID return sagaData, nil }, func(ctx context.Context, data interface{}) error { sagaData := data.(*OrderSagaData) return orderSvc.CancelOrder(ctx, sagaData.OrderID) }, ) // Step 2: Process Payment processPaymentStep := NewStep( "ProcessPayment", func(ctx context.Context, data interface{}) (interface{}, error) { sagaData := data.(*OrderSagaData) var total float64 for _, item := range sagaData.Items { total += item.Price * float64(item.Quantity) } payment, err := paymentSvc.ProcessPayment(ctx, sagaData.OrderID, total) if err != nil { return nil, err } sagaData.PaymentID = payment.ID return sagaData, nil }, func(ctx context.Context, data interface{}) error { sagaData := data.(*OrderSagaData) return paymentSvc.RefundPayment(ctx, sagaData.PaymentID) }, ) // Step 3: Reserve Inventory reserveInventoryStep := NewStep( "ReserveInventory", func(ctx context.Context, data interface{}) (interface{}, error) { sagaData := data.(*OrderSagaData) // Reserve first item (simplified) if len(sagaData.Items) == 0 { return nil, fmt.Errorf("no items to reserve") } item := sagaData.Items[0] reservation, err := inventorySvc.ReserveInventory( ctx, sagaData.OrderID, item.ProductID, item.Quantity, ) if err != nil { return nil, err } sagaData.ReservationID = reservation.ID return sagaData, nil }, func(ctx context.Context, data interface{}) error { sagaData := data.(*OrderSagaData) return inventorySvc.ReleaseReservation(ctx, sagaData.ReservationID) }, ) // Step 4: Schedule Shipment scheduleShipmentStep := NewStep( "ScheduleShipment", func(ctx context.Context, data interface{}) (interface{}, error) { sagaData := data.(*OrderSagaData) shipment, err := shippingSvc.ScheduleShipment(ctx, sagaData.OrderID) if err != nil { return nil, err } sagaData.ShipmentID = shipment.ID return sagaData, nil }, func(ctx context.Context, data interface{}) error { sagaData := data.(*OrderSagaData) return shippingSvc.CancelShipment(ctx, sagaData.ShipmentID) }, ) return []*Step{ createOrderStep, processPaymentStep, reserveInventoryStep, scheduleShipmentStep, } } Main Application // cmd/api/main.go package main import ( "context" "encoding/json" "log" "net/http" "github.com/gorilla/mux" "app/internal/saga" "app/internal/services/inventory" "app/internal/services/order" "app/internal/services/payment" "app/internal/services/shipping" ) func main() { // Initialize services orderSvc := order.NewService() paymentSvc := payment.NewService() inventorySvc := inventory.NewService() shippingSvc := shipping.NewService() // Initialize saga orchestrator orchestrator := saga.NewOrchestrator() router := mux.NewRouter() // Create order endpoint (triggers saga) router.HandleFunc("/orders", func(w http.ResponseWriter, r *http.Request) { var req struct { UserID string `json:"user_id"` Items []order.OrderItem `json:"items"` } if err := json.NewDecoder(r.Body).Decode(&req); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } // Create saga steps := saga.CreateOrderSaga(orderSvc, paymentSvc, inventorySvc, shippingSvc) orderSaga := orchestrator.NewSaga(generateSagaID(), steps) // Execute saga sagaData := &saga.OrderSagaData{ UserID: req.UserID, Items: req.Items, } if err := orchestrator.Execute(context.Background(), orderSaga, sagaData); err != nil { http.Error(w, fmt.Sprintf("Saga failed: %v", err), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusCreated) json.NewEncoder(w).Encode(map[string]string{ "saga_id": orderSaga.ID, "order_id": sagaData.OrderID, "status": string(orderSaga.Status), }) }).Methods("POST") // Get saga status endpoint router.HandleFunc("/sagas/{id}", func(w http.ResponseWriter, r *http.Request) { vars := mux.Vars(r) sagaID := vars["id"] saga, err := orchestrator.GetSaga(sagaID) if err != nil { http.Error(w, err.Error(), http.StatusNotFound) return } response := map[string]interface{}{ "saga_id": saga.ID, "status": saga.Status, "steps": len(saga.Steps), } w.Header().Set("Content-Type", "application/json") json.NewEncoder(w).Encode(response) }).Methods("GET") router.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) log.Println("Saga orchestrator server starting on :8080") log.Fatal(http.ListenAndServe(":8080", router)) } func generateSagaID() string { return fmt.Sprintf("saga_%d", time.Now().UnixNano()) } Testing Saga // internal/saga/order_saga_test.go package saga_test import ( "context" "testing" "app/internal/saga" "app/internal/services/inventory" "app/internal/services/order" "app/internal/services/payment" "app/internal/services/shipping" ) func TestOrderSaga_HappyPath(t *testing.T) { // Initialize services orderSvc := order.NewService() paymentSvc := payment.NewService() inventorySvc := inventory.NewService() shippingSvc := shipping.NewService() // Create saga orchestrator := saga.NewOrchestrator() steps := saga.CreateOrderSaga(orderSvc, paymentSvc, inventorySvc, shippingSvc) orderSaga := orchestrator.NewSaga("test-saga", steps) // Execute saga sagaData := &saga.OrderSagaData{ UserID: "user_123", Items: []order.OrderItem{ {ProductID: "product_1", Quantity: 2, Price: 100.0}, }, } err := orchestrator.Execute(context.Background(), orderSaga, sagaData) if err != nil { t.Fatalf("Saga execution failed: %v", err) } if orderSaga.Status != saga.SagaStatusCompleted { t.Errorf("Expected status %s, got %s", saga.SagaStatusCompleted, orderSaga.Status) } } func TestOrderSaga_PaymentFailure(t *testing.T) { // Initialize services orderSvc := order.NewService() paymentSvc := payment.NewService() inventorySvc := inventory.NewService() shippingSvc := shipping.NewService() // Create saga orchestrator := saga.NewOrchestrator() steps := saga.CreateOrderSaga(orderSvc, paymentSvc, inventorySvc, shippingSvc) orderSaga := orchestrator.NewSaga("test-saga-fail", steps) // Execute saga with amount that will fail sagaData := &saga.OrderSagaData{ UserID: "user_123", Items: []order.OrderItem{ {ProductID: "product_1", Quantity: 2, Price: 10000.0}, // Exceeds limit }, } err := orchestrator.Execute(context.Background(), orderSaga, sagaData) if err == nil { t.Fatal("Expected saga to fail, but it succeeded") } if orderSaga.Status != saga.SagaStatusCompensated { t.Errorf("Expected status %s, got %s", saga.SagaStatusCompensated, orderSaga.Status) } } Best Practices Idempotency: Ensure all operations can be safely retried Compensating Transactions: Design proper rollback logic Timeout Handling: Set timeouts for each saga step State Persistence: Store saga state for recovery Monitoring: Track saga execution and failures Error Handling: Handle partial failures gracefully Testing: Test both success and failure scenarios Documentation: Document saga workflows clearly Common Pitfalls Missing Compensations: Not implementing proper rollback Non-Idempotent Operations: Operations that can’t be retried safely Circular Dependencies: Services depending on each other in compensation Long-Running Sagas: Sagas that hold resources for too long No State Persistence: Losing saga state on failure Ignoring Partial Failures: Not handling scenarios where compensation fails Over-Compensation: Compensating steps that were never executed When to Use Saga Pattern Use When: ...

The Choreography Problem: Hope-Driven Development For years, we’ve built distributed systems using choreography-Service A fires an event into the void, hoping Service B hears it. Service B processes it and fires another event, hoping Service C is listening. When something fails (and it will), we’re left scrambling through logs across multiple services, trying to piece together what happened. This is hope-driven development, and it’s fundamentally broken. Enter Temporal and the concept of Durable Execution-a paradigm shift that replaces hope with guarantees. ...

Backend Communication Current: NATS & JetStream WebRTC All Posts Apache Kafka What is NATS? NATS is a high-performance, cloud-native messaging system designed for microservices, IoT, and edge computing. It provides a simple yet powerful pub/sub model with subject-based addressing, making it ideal for building distributed systems that require fast, reliable communication. ...

Backend Communication Current: gRPC Streaming WebSockets All Posts WebRTC What is gRPC Streaming? gRPC (gRPC Remote Procedure Call) is a high-performance, open-source RPC framework that uses HTTP/2 for transport, Protocol Buffers for serialization, and provides built-in support for streaming. Unlike traditional request-response RPCs, gRPC streaming enables long-lived connections where either party can send multiple messages over time. ...

When building microservices, choosing the right communication protocol is crucial. While REST and event-driven architectures have their place, gRPC offers a compelling alternative with strong typing, efficient binary serialization, and built-in support for streaming. In this guide, we’ll build a complete note-sharing application using gRPC microservices in Go, with a Vue.js frontend connected through a WebSocket gateway. Why gRPC for Microservices? gRPC brings several advantages to microservice architectures: Strong Typing: Protocol Buffers provide type-safe contracts between services Performance: Binary serialization is faster and more compact than JSON Code Generation: Auto-generate client and server code from .proto files Streaming: Built-in support for bidirectional streaming Language Agnostic: Works across many programming languages HTTP/2: Connection multiplexing, header compression, and server push However, browsers can’t make native gRPC calls. That’s where our WebSocket gateway comes in. ...

Building a real-time note-sharing application is a perfect use case for exploring event-driven architecture. In this comprehensive guide, we’ll build a production-ready system with two microservices (User and Note services) that communicate through NATS, delivering instant updates to a Vue.js frontend. Why Event-Driven Architecture? Traditional request-response patterns create tight coupling between services. When your Note service needs to notify users about changes, you don’t want to make synchronous HTTP calls to every service that cares about notes. Event-driven architecture solves this with loose coupling: ...

Go Architecture Patterns Series: ← Previous: Modular Monolith | Series Overview | Next: Event-Driven Architecture → What is Microservices Architecture? Microservices Architecture is an approach where an application is composed of small, independent services that communicate over a network. Each service is self-contained, owns its data, and can be deployed, scaled, and updated independently. Key Principles: Service Independence: Each service is deployed and scaled independently Single Responsibility: Each service handles a specific business capability Decentralized Data: Each service owns its database API-First Design: Services communicate through well-defined APIs Resilience: Services handle failures gracefully Technology Diversity: Services can use different technologies Architecture Overview %%{init: {'theme':'dark', 'themeVariables': {'primaryTextColor':'#e5e7eb','secondaryTextColor':'#e5e7eb','tertiaryTextColor':'#e5e7eb','textColor':'#e5e7eb','nodeTextColor':'#e5e7eb','edgeLabelText':'#e5e7eb','clusterTextColor':'#e5e7eb','actorTextColor':'#e5e7eb'}}}%% graph TB subgraph "Client Layer" Client[Web/Mobile Client] end subgraph "API Gateway" Gateway[API Gateway / Load Balancer] end subgraph "Service Mesh" UserService[User Service] ProductService[Product Service] OrderService[Order Service] PaymentService[Payment Service] NotificationService[Notification Service] end subgraph "Data Layer" UserDB[(User DB)] ProductDB[(Product DB)] OrderDB[(Order DB)] PaymentDB[(Payment DB)] end subgraph "Infrastructure" MessageBroker[Message Broker] ServiceRegistry[Service Discovery] ConfigServer[Config Server] end Client --> Gateway Gateway --> UserService Gateway --> ProductService Gateway --> OrderService Gateway --> PaymentService UserService --> UserDB ProductService --> ProductDB OrderService --> OrderDB PaymentService --> PaymentDB OrderService -.->|HTTP/gRPC| UserService OrderService -.->|HTTP/gRPC| ProductService OrderService -.->|HTTP/gRPC| PaymentService OrderService -.->|Async| MessageBroker PaymentService -.->|Async| MessageBroker NotificationService -.->|Subscribe| MessageBroker UserService -.-> ServiceRegistry ProductService -.-> ServiceRegistry OrderService -.-> ServiceRegistry style UserService fill:#1e3a5f,color:#fff style ProductService fill:#78350f,color:#fff style OrderService fill:#134e4a,color:#fff style PaymentService fill:#4c1d95,color:#fff style NotificationService fill:#4a1e3a,color:#fff Service Communication Patterns %%{init: {'theme':'dark', 'themeVariables': {'primaryTextColor':'#e5e7eb','secondaryTextColor':'#e5e7eb','tertiaryTextColor':'#e5e7eb','textColor':'#e5e7eb','nodeTextColor':'#e5e7eb','edgeLabelText':'#e5e7eb','clusterTextColor':'#e5e7eb','actorTextColor':'#e5e7eb'}}}%% sequenceDiagram participant Client participant Gateway participant OrderSvc as Order Service participant UserSvc as User Service participant ProductSvc as Product Service participant PaymentSvc as Payment Service participant Queue as Message Queue participant NotifySvc as Notification Service Client->>Gateway: Create Order Request Gateway->>OrderSvc: POST /orders OrderSvc->>UserSvc: GET /users/{id} UserSvc-->>OrderSvc: User Data OrderSvc->>ProductSvc: GET /products/{id} ProductSvc-->>OrderSvc: Product Data OrderSvc->>ProductSvc: POST /products/reserve ProductSvc-->>OrderSvc: Stock Reserved OrderSvc->>PaymentSvc: POST /payments PaymentSvc-->>OrderSvc: Payment Success OrderSvc->>Queue: Publish OrderCreated Event Queue->>NotifySvc: OrderCreated Event NotifySvc->>NotifySvc: Send Email/SMS OrderSvc-->>Gateway: Order Created Gateway-->>Client: Response Real-World Use Cases E-commerce Platforms: Amazon, eBay with separate services for products, orders, payments Streaming Services: Netflix with services for recommendations, playback, billing Ride-Sharing Apps: Uber with services for riders, drivers, payments, routing Financial Systems: Banking apps with separate services for accounts, transactions, loans Social Media: Twitter with services for posts, timelines, notifications, messages Cloud Platforms: AWS-like platforms with independent service offerings Microservices Implementation Project Structure (Multi-Repository) microservices/ ├── user-service/ │ ├── cmd/ │ │ └── server/ │ │ └── main.go │ ├── internal/ │ │ ├── domain/ │ │ ├── handlers/ │ │ ├── repository/ │ │ └── service/ │ ├── proto/ │ │ └── user.proto │ └── go.mod ├── product-service/ │ ├── cmd/ │ │ └── server/ │ │ └── main.go │ ├── internal/ │ │ ├── domain/ │ │ ├── handlers/ │ │ ├── repository/ │ │ └── service/ │ └── go.mod ├── order-service/ │ ├── cmd/ │ │ └── server/ │ │ └── main.go │ ├── internal/ │ │ ├── domain/ │ │ ├── handlers/ │ │ ├── repository/ │ │ ├── service/ │ │ └── clients/ │ └── go.mod └── api-gateway/ ├── cmd/ │ └── server/ │ └── main.go └── go.mod Service 1: User Service // user-service/internal/domain/user.go package domain import ( "context" "errors" "time" ) type User struct { ID string `json:"id"` Email string `json:"email"` Name string `json:"name"` Active bool `json:"active"` CreatedAt time.Time `json:"created_at"` } var ( ErrUserNotFound = errors.New("user not found") ErrUserExists = errors.New("user already exists") ) type Repository interface { Create(ctx context.Context, user *User) error GetByID(ctx context.Context, id string) (*User, error) GetByEmail(ctx context.Context, email string) (*User, error) Update(ctx context.Context, user *User) error } // user-service/internal/service/user_service.go package service import ( "context" "fmt" "time" "user-service/internal/domain" ) type UserService struct { repo domain.Repository } func NewUserService(repo domain.Repository) *UserService { return &UserService{repo: repo} } func (s *UserService) CreateUser(ctx context.Context, email, name string) (*domain.User, error) { existing, _ := s.repo.GetByEmail(ctx, email) if existing != nil { return nil, domain.ErrUserExists } user := &domain.User{ ID: generateID(), Email: email, Name: name, Active: true, CreatedAt: time.Now(), } if err := s.repo.Create(ctx, user); err != nil { return nil, fmt.Errorf("failed to create user: %w", err) } return user, nil } func (s *UserService) GetUser(ctx context.Context, id string) (*domain.User, error) { return s.repo.GetByID(ctx, id) } func (s *UserService) ValidateUser(ctx context.Context, id string) (bool, error) { user, err := s.repo.GetByID(ctx, id) if err != nil { return false, err } return user.Active, nil } func generateID() string { return fmt.Sprintf("user_%d", time.Now().UnixNano()) } // user-service/internal/handlers/http_handler.go package handlers import ( "encoding/json" "net/http" "github.com/gorilla/mux" "user-service/internal/service" ) type HTTPHandler struct { service *service.UserService } func NewHTTPHandler(service *service.UserService) *HTTPHandler { return &HTTPHandler{service: service} } type CreateUserRequest struct { Email string `json:"email"` Name string `json:"name"` } func (h *HTTPHandler) CreateUser(w http.ResponseWriter, r *http.Request) { var req CreateUserRequest if err := json.NewDecoder(r.Body).Decode(&req); err != nil { respondError(w, http.StatusBadRequest, "invalid request") return } user, err := h.service.CreateUser(r.Context(), req.Email, req.Name) if err != nil { respondError(w, http.StatusBadRequest, err.Error()) return } respondJSON(w, http.StatusCreated, user) } func (h *HTTPHandler) GetUser(w http.ResponseWriter, r *http.Request) { vars := mux.Vars(r) id := vars["id"] user, err := h.service.GetUser(r.Context(), id) if err != nil { respondError(w, http.StatusNotFound, "user not found") return } respondJSON(w, http.StatusOK, user) } func (h *HTTPHandler) ValidateUser(w http.ResponseWriter, r *http.Request) { vars := mux.Vars(r) id := vars["id"] valid, err := h.service.ValidateUser(r.Context(), id) if err != nil { respondError(w, http.StatusNotFound, "user not found") return } respondJSON(w, http.StatusOK, map[string]bool{"valid": valid}) } func respondJSON(w http.ResponseWriter, status int, data interface{}) { w.Header().Set("Content-Type", "application/json") w.WriteHeader(status) json.NewEncoder(w).Encode(data) } func respondError(w http.ResponseWriter, status int, message string) { respondJSON(w, status, map[string]string{"error": message}) } // user-service/cmd/server/main.go package main import ( "database/sql" "log" "net/http" "os" "github.com/gorilla/mux" _ "github.com/lib/pq" "user-service/internal/handlers" "user-service/internal/repository" "user-service/internal/service" ) func main() { dbURL := os.Getenv("DATABASE_URL") if dbURL == "" { dbURL = "postgres://user:pass@localhost/users?sslmode=disable" } db, err := sql.Open("postgres", dbURL) if err != nil { log.Fatal(err) } defer db.Close() repo := repository.NewPostgresRepository(db) svc := service.NewUserService(repo) handler := handlers.NewHTTPHandler(svc) router := mux.NewRouter() router.HandleFunc("/users", handler.CreateUser).Methods("POST") router.HandleFunc("/users/{id}", handler.GetUser).Methods("GET") router.HandleFunc("/users/{id}/validate", handler.ValidateUser).Methods("GET") router.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) w.Write([]byte("OK")) }) port := os.Getenv("PORT") if port == "" { port = "8081" } log.Printf("User service starting on port %s", port) if err := http.ListenAndServe(":"+port, router); err != nil { log.Fatal(err) } } Service 2: Product Service // product-service/internal/domain/product.go package domain import ( "context" "errors" "time" ) type Product struct { ID string `json:"id"` Name string `json:"name"` Description string `json:"description"` Price float64 `json:"price"` Stock int `json:"stock"` CreatedAt time.Time `json:"created_at"` } var ( ErrProductNotFound = errors.New("product not found") ErrInsufficientStock = errors.New("insufficient stock") ) type Repository interface { Create(ctx context.Context, product *Product) error GetByID(ctx context.Context, id string) (*Product, error) Update(ctx context.Context, product *Product) error ReserveStock(ctx context.Context, id string, quantity int) error } // product-service/internal/service/product_service.go package service import ( "context" "fmt" "time" "product-service/internal/domain" ) type ProductService struct { repo domain.Repository } func NewProductService(repo domain.Repository) *ProductService { return &ProductService{repo: repo} } func (s *ProductService) CreateProduct(ctx context.Context, name, desc string, price float64, stock int) (*domain.Product, error) { product := &domain.Product{ ID: generateID(), Name: name, Description: desc, Price: price, Stock: stock, CreatedAt: time.Now(), } if err := s.repo.Create(ctx, product); err != nil { return nil, fmt.Errorf("failed to create product: %w", err) } return product, nil } func (s *ProductService) GetProduct(ctx context.Context, id string) (*domain.Product, error) { return s.repo.GetByID(ctx, id) } func (s *ProductService) ReserveStock(ctx context.Context, id string, quantity int) error { product, err := s.repo.GetByID(ctx, id) if err != nil { return err } if product.Stock < quantity { return domain.ErrInsufficientStock } return s.repo.ReserveStock(ctx, id, quantity) } func generateID() string { return fmt.Sprintf("product_%d", time.Now().UnixNano()) } // product-service/cmd/server/main.go package main import ( "database/sql" "encoding/json" "log" "net/http" "os" "github.com/gorilla/mux" _ "github.com/lib/pq" "product-service/internal/repository" "product-service/internal/service" ) func main() { dbURL := os.Getenv("DATABASE_URL") if dbURL == "" { dbURL = "postgres://user:pass@localhost/products?sslmode=disable" } db, err := sql.Open("postgres", dbURL) if err != nil { log.Fatal(err) } defer db.Close() repo := repository.NewPostgresRepository(db) svc := service.NewProductService(repo) router := mux.NewRouter() router.HandleFunc("/products/{id}", func(w http.ResponseWriter, r *http.Request) { vars := mux.Vars(r) product, err := svc.GetProduct(r.Context(), vars["id"]) if err != nil { http.Error(w, err.Error(), http.StatusNotFound) return } json.NewEncoder(w).Encode(product) }).Methods("GET") router.HandleFunc("/products/reserve", func(w http.ResponseWriter, r *http.Request) { var req struct { ProductID string `json:"product_id"` Quantity int `json:"quantity"` } if err := json.NewDecoder(r.Body).Decode(&req); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } if err := svc.ReserveStock(r.Context(), req.ProductID, req.Quantity); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } w.WriteHeader(http.StatusOK) json.NewEncoder(w).Encode(map[string]string{"status": "reserved"}) }).Methods("POST") router.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) port := os.Getenv("PORT") if port == "" { port = "8082" } log.Printf("Product service starting on port %s", port) if err := http.ListenAndServe(":"+port, router); err != nil { log.Fatal(err) } } Service 3: Order Service (Orchestrator) // order-service/internal/clients/user_client.go package clients import ( "context" "encoding/json" "fmt" "net/http" "time" ) type UserClient struct { baseURL string httpClient *http.Client } func NewUserClient(baseURL string) *UserClient { return &UserClient{ baseURL: baseURL, httpClient: &http.Client{ Timeout: 5 * time.Second, }, } } func (c *UserClient) ValidateUser(ctx context.Context, userID string) (bool, error) { url := fmt.Sprintf("%s/users/%s/validate", c.baseURL, userID) req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return false, err } resp, err := c.httpClient.Do(req) if err != nil { return false, fmt.Errorf("failed to call user service: %w", err) } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { return false, fmt.Errorf("user service returned status %d", resp.StatusCode) } var result struct { Valid bool `json:"valid"` } if err := json.NewDecoder(resp.Body).Decode(&result); err != nil { return false, err } return result.Valid, nil } // order-service/internal/clients/product_client.go package clients import ( "bytes" "context" "encoding/json" "fmt" "net/http" "time" ) type Product struct { ID string `json:"id"` Name string `json:"name"` Price float64 `json:"price"` Stock int `json:"stock"` } type ProductClient struct { baseURL string httpClient *http.Client } func NewProductClient(baseURL string) *ProductClient { return &ProductClient{ baseURL: baseURL, httpClient: &http.Client{ Timeout: 5 * time.Second, }, } } func (c *ProductClient) GetProduct(ctx context.Context, productID string) (*Product, error) { url := fmt.Sprintf("%s/products/%s", c.baseURL, productID) req, err := http.NewRequestWithContext(ctx, "GET", url, nil) if err != nil { return nil, err } resp, err := c.httpClient.Do(req) if err != nil { return nil, fmt.Errorf("failed to call product service: %w", err) } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { return nil, fmt.Errorf("product not found") } var product Product if err := json.NewDecoder(resp.Body).Decode(&product); err != nil { return nil, err } return &product, nil } func (c *ProductClient) ReserveStock(ctx context.Context, productID string, quantity int) error { url := fmt.Sprintf("%s/products/reserve", c.baseURL) reqBody := map[string]interface{}{ "product_id": productID, "quantity": quantity, } body, err := json.Marshal(reqBody) if err != nil { return err } req, err := http.NewRequestWithContext(ctx, "POST", url, bytes.NewBuffer(body)) if err != nil { return err } req.Header.Set("Content-Type", "application/json") resp, err := c.httpClient.Do(req) if err != nil { return fmt.Errorf("failed to reserve stock: %w", err) } defer resp.Body.Close() if resp.StatusCode != http.StatusOK { return fmt.Errorf("failed to reserve stock: status %d", resp.StatusCode) } return nil } // order-service/internal/service/order_service.go package service import ( "context" "fmt" "time" "order-service/internal/clients" "order-service/internal/domain" ) type OrderService struct { repo domain.Repository userClient *clients.UserClient productClient *clients.ProductClient } func NewOrderService( repo domain.Repository, userClient *clients.UserClient, productClient *clients.ProductClient, ) *OrderService { return &OrderService{ repo: repo, userClient: userClient, productClient: productClient, } } func (s *OrderService) CreateOrder(ctx context.Context, userID string, items []domain.OrderItem) (*domain.Order, error) { // Validate user via User Service valid, err := s.userClient.ValidateUser(ctx, userID) if err != nil { return nil, fmt.Errorf("failed to validate user: %w", err) } if !valid { return nil, fmt.Errorf("user is not valid") } // Validate products and calculate total var total float64 for i, item := range items { product, err := s.productClient.GetProduct(ctx, item.ProductID) if err != nil { return nil, fmt.Errorf("failed to get product: %w", err) } items[i].Price = product.Price total += product.Price * float64(item.Quantity) } // Reserve stock via Product Service for _, item := range items { if err := s.productClient.ReserveStock(ctx, item.ProductID, item.Quantity); err != nil { return nil, fmt.Errorf("failed to reserve stock: %w", err) } } order := &domain.Order{ ID: generateID(), UserID: userID, Items: items, Total: total, Status: "pending", CreatedAt: time.Now(), } if err := s.repo.Create(ctx, order); err != nil { return nil, fmt.Errorf("failed to create order: %w", err) } return order, nil } func generateID() string { return fmt.Sprintf("order_%d", time.Now().UnixNano()) } // order-service/cmd/server/main.go package main import ( "database/sql" "encoding/json" "log" "net/http" "os" "github.com/gorilla/mux" _ "github.com/lib/pq" "order-service/internal/clients" "order-service/internal/domain" "order-service/internal/repository" "order-service/internal/service" ) func main() { dbURL := os.Getenv("DATABASE_URL") if dbURL == "" { dbURL = "postgres://user:pass@localhost/orders?sslmode=disable" } userServiceURL := os.Getenv("USER_SERVICE_URL") if userServiceURL == "" { userServiceURL = "http://localhost:8081" } productServiceURL := os.Getenv("PRODUCT_SERVICE_URL") if productServiceURL == "" { productServiceURL = "http://localhost:8082" } db, err := sql.Open("postgres", dbURL) if err != nil { log.Fatal(err) } defer db.Close() repo := repository.NewPostgresRepository(db) userClient := clients.NewUserClient(userServiceURL) productClient := clients.NewProductClient(productServiceURL) svc := service.NewOrderService(repo, userClient, productClient) router := mux.NewRouter() router.HandleFunc("/orders", func(w http.ResponseWriter, r *http.Request) { var req struct { UserID string `json:"user_id"` Items []domain.OrderItem `json:"items"` } if err := json.NewDecoder(r.Body).Decode(&req); err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } order, err := svc.CreateOrder(r.Context(), req.UserID, req.Items) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } w.Header().Set("Content-Type", "application/json") w.WriteHeader(http.StatusCreated) json.NewEncoder(w).Encode(order) }).Methods("POST") router.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) port := os.Getenv("PORT") if port == "" { port = "8083" } log.Printf("Order service starting on port %s", port) if err := http.ListenAndServe(":"+port, router); err != nil { log.Fatal(err) } } Docker Compose Setup version: '3.8' services: user-service: build: ./user-service ports: - "8081:8081" environment: - DATABASE_URL=postgres://user:pass@user-db:5432/users?sslmode=disable - PORT=8081 depends_on: - user-db product-service: build: ./product-service ports: - "8082:8082" environment: - DATABASE_URL=postgres://user:pass@product-db:5432/products?sslmode=disable - PORT=8082 depends_on: - product-db order-service: build: ./order-service ports: - "8083:8083" environment: - DATABASE_URL=postgres://user:pass@order-db:5432/orders?sslmode=disable - USER_SERVICE_URL=http://user-service:8081 - PRODUCT_SERVICE_URL=http://product-service:8082 - PORT=8083 depends_on: - order-db - user-service - product-service user-db: image: postgres:15 environment: - POSTGRES_USER=user - POSTGRES_PASSWORD=pass - POSTGRES_DB=users product-db: image: postgres:15 environment: - POSTGRES_USER=user - POSTGRES_PASSWORD=pass - POSTGRES_DB=products order-db: image: postgres:15 environment: - POSTGRES_USER=user - POSTGRES_PASSWORD=pass - POSTGRES_DB=orders Best Practices Service Boundaries: Define clear service boundaries based on business capabilities API Contracts: Use API versioning and maintain backward compatibility Service Discovery: Implement service registry for dynamic service location Circuit Breakers: Prevent cascading failures with circuit breaker pattern Distributed Tracing: Implement tracing to debug cross-service calls Health Checks: Provide health endpoints for monitoring Configuration Management: Externalize configuration Security: Implement service-to-service authentication Common Pitfalls Distributed Monolith: Services too tightly coupled, defeating the purpose Chatty Services: Too many synchronous calls between services Shared Database: Multiple services accessing the same database Ignoring Network Failures: Not handling network errors gracefully No Service Versioning: Breaking changes without versioning Data Consistency Issues: Not handling eventual consistency Over-Engineering: Creating too many small services When to Use Microservices Architecture Use When: ...