You are viewing documentation for an older version of Lucille.

This is a static snapshot.
For up-to-date information, see the latest version.

Stage

A Stage performs a specific transformation on a Document.

What a Stage Does

A Stage is the fundamental unit of document transformation in Lucille. Each Stage performs a single, focused operation on a Document: extracting text, renaming fields, generating embeddings, looking up data from an external system, or any other enrichment task.

Stages are composed into Pipelines. When a Document flows through a Pipeline, it passes through each Stage in sequence. Each Stage receives the Document as mutated by all previous Stages and can read fields, write fields, or emit child documents.

The Stage Contract

A Stage implementation must provide one method: processDocument(Document doc). This method receives a Document, performs its transformation (typically by reading and writing fields), and returns an iterator of result documents. For most stages, the iterator contains just the input document (now modified). Stages that generate child documents return the children followed by the parent.

The framework handles everything else:

  • Instantiation — Stages are created from class names in configuration via reflection.
  • Lifecycle — start() is called once before processing begins (for resource acquisition); stop() is called once after processing ends (for cleanup).
  • Condition evaluation — The framework checks conditions before calling processDocument(). If conditions are not met, the Stage is skipped entirely.
  • Thread isolation — Each Worker thread gets its own Stage instance. No synchronization is needed.
  • Error handling — If processDocument() throws, the framework catches the exception, marks the document as failed, and continues processing other documents.

Conditions as a Design Decision

Rather than supporting sub-pipelines or branching, Lucille provides per-stage conditions that control whether a Stage applies to a given Document. This keeps the pipeline linear while allowing different processing for different document types.

Conditions are evaluated by the framework before invoking the Stage. A Stage author never implements conditional logic — they write a Stage that does one thing, and the configuration determines which documents it applies to. This separation means Stages are simpler to write, simpler to test, and reusable across pipelines with different condition configurations.

Disabling Stages

As a convenience, you can set enabled: false on any Stage in your Config. By doing so, the Stage is not instantiated, start() and stop() are never called, and no Document is processed by that Stage. The Stage will still be validated against its Spec, warning you of any missing, invalid, or unknown properties in the Config.

Child Document Emission

A Stage can produce additional documents — children — that flow through the remaining pipeline stages independently. This is how Lucille handles 1-to-N fan-out (e.g., chunking a document into embedding-sized pieces). Children are tracked by the Publisher’s accounting system and indexed as independent records.

The iterator-based return type (Iterator<Document>) means children are produced lazily. Memory usage is bounded regardless of how many children a Stage generates.

Practical Guide

For how to configure stages — syntax, conditions, conditionPolicy, and the full catalogue of built-in stages — see Stages in the Ingest Designer Guide.

For how to build a custom Stage, see Developing Stages.