Distributed-Systems

Introduction Apache Kafka is the backbone of event-driven architectures at companies like LinkedIn, Uber, and Airbnb. It lets services communicate asynchronously at massive scale — without coupling them together. Go is an excellent fit for Kafka workloads: lightweight goroutines, clean channel semantics, and minimal overhead make it easy to build high-throughput producers and consumers. This guide skips the theory-heavy basics and focuses on what you actually build. We’ll cover: Kafka’s core model in plain terms Setting up producers and consumers with kafka-go Three real-world use cases with full code and diagrams Patterns for reliability: retries, dead-letter queues, and consumer groups How Kafka Works (The Short Version) Kafka organizes data into topics. Each topic is split into partitions — ordered, immutable logs. Producers append messages to partitions. Consumers read messages from partitions at their own pace, tracking their position with an offset. ...

How to Read This Post Each scenario shows a diagram first, then a short note on why that system fits best. Complexity increases as you scroll. System Strength RabbitMQ Smart broker, routing flexibility, push-based delivery Kafka Immutable log, high throughput, replay from any offset JetStream NATS Lightweight persistence, built-in dedup, low operational cost Level 1 — Foundations 1. Simple Task Queue flowchart LR P[Producer] -->|task| Q[(Queue)] Q -->|deliver| C[Consumer] style P fill:#4a9eff,stroke:#2d7ed8,color:#fff style Q fill:#ff6b6b,stroke:#d44,color:#fff style C fill:#51cf66,stroke:#37b24d,color:#fff RabbitMQ. Push-based delivery with acknowledgments. Consumer confirms completion before the broker removes the message. No polling, no offset tracking — just send and ack. ...

How to Read This Post This post is a visual reference for caching strategies, organized by increasing complexity. Each section contains a Mermaid diagram followed by a brief explanation. No code — just patterns and pictures. Strategy Type Consistency Latency Complexity Best For Cache-Aside Read Eventual Low reads Low General purpose Read-Through Read Eventual Low reads Medium Read-heavy workloads Write-Through Write Strong High writes Medium Consistency-critical writes Write-Behind Write Eventual Low writes High Write-heavy workloads Write-Around Write Eventual Low writes Low Rarely-read written data LRU / LFU / TTL Eviction N/A N/A Low–Medium Memory management Consistent Hashing Distribution Varies Low High Horizontal scaling Multi-Layer Cache Read Eventual Very low High High-traffic systems Level 1 — Foundations 1. No Cache Baseline %%{init: {'theme':'dark', 'themeVariables': {'primaryTextColor':'#e5e7eb','secondaryTextColor':'#e5e7eb','tertiaryTextColor':'#e5e7eb','textColor':'#e5e7eb','nodeTextColor':'#e5e7eb','edgeLabelText':'#e5e7eb','clusterTextColor':'#e5e7eb','actorTextColor':'#e5e7eb'}}}%% flowchart LR subgraph Clients C1([Client 1]):::blue C2([Client 2]):::blue C3([Client 3]):::blue end subgraph Application A[App Server]:::purple end subgraph Storage DB[(Database)]:::red end C1 -->|Request| A C2 -->|Request| A C3 -->|Request| A A -->|"Query (50ms)"| DB DB -->|"Response (50ms)"| A A -->|"Total: ~100ms"| C1 A -->|"Total: ~100ms"| C2 A -->|"Total: ~100ms"| C3 classDef blue fill:#4a9eff,stroke:#4a9eff,color:#e5e7eb classDef red fill:#ff6b6b,stroke:#ff6b6b,color:#e5e7eb classDef purple fill:#a29bfe,stroke:#a29bfe,color:#e5e7eb No Cache Baseline — Every request hits the database directly. Under load, the database becomes the bottleneck: latency climbs, connections exhaust, and throughput collapses. This is the problem caching solves. ...

How to Read This Post Each scenario shows a diagram first, then a short note on why the pattern matters. Complexity increases as you scroll. Pattern Approach Consistency Best For 2PC Coordinator-driven Strong (atomic) Short-lived, cross-DB 3PC Non-blocking 2PC Strong (reduces blocking) Theoretical improvement Saga Choreography Event-driven chain Eventual Loosely coupled services Saga Orchestration Central coordinator Eventual Complex workflows Outbox Pattern Local TX + relay At-least-once Event publishing guarantee Level 1 — Foundations 1. The Distributed Transaction Problem %%{init: {'theme':'dark', 'themeVariables': {'primaryTextColor':'#e5e7eb','secondaryTextColor':'#e5e7eb','tertiaryTextColor':'#e5e7eb','textColor':'#e5e7eb','nodeTextColor':'#e5e7eb','edgeLabelText':'#e5e7eb','clusterTextColor':'#e5e7eb','actorTextColor':'#e5e7eb'}}}%% flowchart TB C["Client: Place Order"] C --> OS["Order ServiceCreate order ✓"] C --> PS["Payment ServiceCharge $50 ✓"] C --> IS["Inventory ServiceReserve item ✗ FAILED"] IS -->|"What now?"| PROBLEM{{"Order created,payment charged,but no inventory!"}} PROBLEM --> Q1["Refund payment?"] PROBLEM --> Q2["Cancel order?"] PROBLEM --> Q3["Retry inventory?"] PROBLEM --> Q4["All of the abovein what order?"] style C fill:#4a9eff,stroke:#2d7ed8,color:#fff style OS fill:#51cf66,stroke:#37b24d,color:#fff style PS fill:#51cf66,stroke:#37b24d,color:#fff style IS fill:#ff6b6b,stroke:#d44,color:#fff style PROBLEM fill:#ffd43b,stroke:#f59f00,color:#333 style Q1 fill:#546e7a,stroke:#90a4ae,color:#fff style Q2 fill:#546e7a,stroke:#90a4ae,color:#fff style Q3 fill:#546e7a,stroke:#90a4ae,color:#fff style Q4 fill:#a29bfe,stroke:#6c5ce7,color:#fff The core problem. In a monolith, a single database transaction guarantees all-or-nothing. In microservices, each service has its own database — there’s no single transaction boundary. If one step fails after others succeed, you have an inconsistent state. ...

How to Read This Post Each scenario shows a diagram first, then a short note on why the pattern matters. Complexity increases as you scroll. Strategy Category Strength Trade-off Leader-Follower Replication Simple, strong consistency for reads Write bottleneck at leader Multi-Leader Replication Multi-DC writes Conflict resolution needed Leaderless (Quorum) Replication No single point of failure Tunable consistency Range Partitioning Partitioning Range scans, sorted data Hotspot risk Hash Partitioning Partitioning Even distribution No range queries Consistent Hashing Partitioning Minimal redistribution Virtual nodes needed Level 1 — Foundations 1. Single Node Storage %%{init: {'theme':'dark', 'themeVariables': {'primaryTextColor':'#e5e7eb','secondaryTextColor':'#e5e7eb','tertiaryTextColor':'#e5e7eb','textColor':'#e5e7eb','nodeTextColor':'#e5e7eb','edgeLabelText':'#e5e7eb','clusterTextColor':'#e5e7eb','actorTextColor':'#e5e7eb'}}}%% flowchart LR C1["Client 1"] --> DB[(Single Database)] C2["Client 2"] --> DB C3["Client 3"] --> DB DB -->|"⚡ crash"| LOST["All data LOSTAll clients BLOCKED"] style C1 fill:#4a9eff,stroke:#2d7ed8,color:#fff style C2 fill:#4a9eff,stroke:#2d7ed8,color:#fff style C3 fill:#4a9eff,stroke:#2d7ed8,color:#fff style DB fill:#ff6b6b,stroke:#d44,color:#fff style LOST fill:#546e7a,stroke:#90a4ae,color:#fff Single point of failure. One database handles everything — simple but fatal. If it crashes, all data is lost and all clients are blocked. Every distributed storage pattern exists to solve this problem. ...

How to Read This Post Each scenario shows a diagram first, then a short note on why the pattern matters. Complexity increases as you scroll. Algorithm Approach Strength Trade-off Raft Leader-based Understandable, widely adopted Leader bottleneck Paxos Proposer-based Formally proven, flexible Notoriously hard to implement Multi-Paxos Optimized Paxos Fewer round-trips with stable leader Complex recovery ZAB Leader-based Ordered broadcast (ZooKeeper) Tightly coupled to ZK Level 1 — Why Consensus? 1. The Agreement Problem %%{init: {'theme':'dark', 'themeVariables': {'primaryTextColor':'#e5e7eb','secondaryTextColor':'#e5e7eb','tertiaryTextColor':'#e5e7eb','textColor':'#e5e7eb','nodeTextColor':'#e5e7eb','edgeLabelText':'#e5e7eb','clusterTextColor':'#e5e7eb','actorTextColor':'#e5e7eb'}}}%% flowchart TB C[Client] -->|"SET x=5"| N1["Node Ax = 5 ✓"] C -->|"SET x=5"| N2["Node Bx = 5 ✓"] C -->|"SET x=5"| N3["Node C⚡ crashed"] N1 -->|"what is x?"| Q{{"Do all nodesagree x=5?"}} N2 -->|"what is x?"| Q N3 -->|"???"| Q Q -->|"With consensus"| YES["All agree on x=5even with failures"] Q -->|"Without consensus"| NO["Inconsistent statesplit brain possible"] style C fill:#4a9eff,stroke:#2d7ed8,color:#fff style N1 fill:#51cf66,stroke:#37b24d,color:#fff style N2 fill:#51cf66,stroke:#37b24d,color:#fff style N3 fill:#ff6b6b,stroke:#d44,color:#fff style Q fill:#ffd43b,stroke:#f59f00,color:#333 style YES fill:#51cf66,stroke:#37b24d,color:#fff style NO fill:#ff6b6b,stroke:#d44,color:#fff The core problem. Distributed nodes must agree on a single value even when some nodes crash or messages are delayed. Without consensus, you get split-brain — two parts of the system believing different things. ...

Building reliable distributed systems is challenging. Network failures, service outages, and unexpected errors can leave your workflows in inconsistent states. What if there was a way to build workflows that are inherently resilient, automatically handling retries, timeouts, and state management? Enter Temporal.io - a workflow orchestration platform that makes building reliable distributed applications dramatically easier. In this comprehensive tutorial, we’ll build a coffee shop ordering system that demonstrates Temporal’s powerful capabilities. ...

The Token Ring Problem The Token Ring is a classic distributed mutual exclusion algorithm where nodes are arranged in a logical ring, and a single token circulates. Only the node holding the token can access the shared resource. It’s simple, fair, and starvation-free. The Scenario Nodes arranged in a ring: N nodes form a logical ring A single token passes around the ring Only token holder can enter critical section After using resource, pass token to next node Properties: Fair (FIFO order), no starvation, deadlock-free The challenge: ...

The Gossiping Problem The Gossiping Problem is a classic problem in distributed systems and graph theory. N people each know a unique secret, and they share information through phone calls. On each call, both parties share all secrets they know. The goal: find the minimum number of calls needed for everyone to know all secrets. The Scenario N people, N secrets: Person i knows secret Si initially When persons i and j call each other, they share ALL secrets they know Goal: Everyone knows all N secrets Question: What’s the minimum number of calls? The Mathematical Result For n ≥ 4 people: ...

In the world of modern software development, the question isn’t whether you’ll need to scale-it’s when. If you’ve ever watched a monolithic application groan under increasing load, fought to deploy a single feature without breaking everything else, or felt trapped by technology choices made years ago, you’re not alone. Let’s explore how event-driven microservices in Go can solve these challenges and build systems that scale gracefully with your ambitions. The Pain of the Monolith Picture this: Your application has grown from a simple CRUD app to a complex beast handling users, notes, notifications, analytics, and more. Every deployment is a nail-biting experience because changing one module might break three others. Your database has become a bottleneck, and adding more servers doesn’t help because everything shares the same database connection pool. Different teams step on each other’s toes, and that cool new technology? Sorry, the entire stack is locked into decisions made in 2015. ...