https://kmwtechnology.github.io/lucille/docs-0.10.0/architecture/components/pipeline/Recent content in Pipeline on LucilleHugoenMon, 09 Jun 2025 00:00:00 +0000https://kmwtechnology.github.io/lucille/docs-0.10.0/architecture/components/pipeline/pipeline-internals/Mon, 09 Jun 2025 00:00:00 +0000https://kmwtechnology.github.io/lucille/docs-0.10.0/architecture/components/pipeline/pipeline-internals/<h2 id="what-a-pipeline-is">What a Pipeline Is</h2> <p>A Pipeline is an ordered sequence of Stages. When a Document enters the Pipeline, it flows through each Stage in order. Each Stage modifies the Document in place and may optionally generate child Documents. The Pipeline returns an Iterator over all Documents that emerged from the processing — the original (possibly modified) Document plus any children generated along the way.</p> <p>The key insight is that the Pipeline does not eagerly process everything and return a collection. It returns a <strong>lazy Iterator</strong> that processes Documents on demand as <code>next()</code> is called. This has profound implications for memory usage, especially when stages generate children.</p>