https://alamrafiul.com/categories/patterns/ Recent content in Patterns on Rafiul Alam https://alamrafiul.com/papermod-cover.png https://alamrafiul.com/papermod-cover.png Hugo -- gohugo.io en Fri, 30 Aug 2024 00:00:00 +0000 https://alamrafiul.com/posts/go-pipeline-streaming/ Fri, 30 Aug 2024 00:00:00 +0000 https://alamrafiul.com/posts/go-pipeline-streaming/ <h2 id="the-power-of-streaming-pipelines">The Power of Streaming Pipelines</h2> <p>Imagine processing a million log entries. The naive approach loads everything into memory, processes it, then outputs results. But what if you don&rsquo;t have enough RAM? What if you want results streaming in real-time?</p> <p>Pipeline patterns break complex processing into stages connected by channels. Data flows through the pipeline, with each stage transforming it concurrently. It&rsquo;s Unix pipes meets goroutines-and it&rsquo;s beautiful.</p> <h3 id="the-sequential-approach">The Sequential Approach</h3> <p>Here&rsquo;s what we&rsquo;re moving away from:</p>