Stages

For conceptual documentation — what a Stage is, the Stage contract, conditions as a design decision, and child document emission — see Architecture: Stage.

Configuring a Stage

To configure a Stage, provide its class in the config. You can also specify a name (for logging and error messages), conditions, and conditionPolicy:

{
  name: "AddRandomBoolean-First"
  class: "com.kmwllc.lucille.stage.AddRandomBoolean"
  field_name: "rand_bool_1"
  percent_true: 65
}

Each Stage also accepts its own implementation-specific parameters (like field_name and percent_true above). See the individual stage pages below for details.

Conditions

For any Stage, you can specify conditions in its config to control when the Stage processes a Document.

Condition parameters

values and valuesPath are mutually exclusive — specifying both is an error.

How matching works

With values or valuesPath: The condition passes if any of the listed fields contains any of the listed values. Matching is type-coerced to string — a boolean field true matches the value "true", an integer 10 matches "10". null is a valid value entry and will match a null field value.

Without values or valuesPath: The condition checks field existence only.

• operator: "must" — passes if all listed fields are present on the document.
• operator: "must_not" — passes if all listed fields are absent from the document.

conditionPolicy

When a stage has multiple conditions, conditionPolicy in the stage’s root config controls how they combine:

• "all" (default) — all conditions must be met
• "any" — at least one condition must be met

Examples

Run a stage only when a field exists:

{
  class: "com.kmwllc.lucille.stage.MyStage"
  conditions: [
    { fields: ["content"] }
  ]
}

Run a stage only when a field matches a value:

{
  name: "print-1"
  class: "com.kmwllc.lucille.stage.Print"
  conditions: [
    { fields: ["city"], values: ["Boston", "New York"] }
  ]
}

Skip a stage when a field is present (must_not existence check):

{
  class: "com.kmwllc.lucille.stage.OpenAIEmbed"
  conditions: [
    { fields: ["embedding"], operator: "must_not" }
  ]
}

Require multiple conditions (all must be met):

{
  class: "com.kmwllc.lucille.stage.OpenAIEmbed"
  conditionPolicy: "all"
  conditions: [
    { fields: ["content"] }
    { fields: ["content_type"], values: ["article"] }
  ]
}

Load match values from a file:

{
  class: "com.kmwllc.lucille.stage.DropDocument"
  conditions: [
    { fields: ["category"], valuesPath: "s3://my-bucket/excluded-categories.txt" }
  ]
}

For the full reference on controlling document fate and connector sequencing — conditions, skipping, dropping, error handling, child documents, and more — see Control Flow.

Stage Catalogue

See All Stages for a complete listing of all available stages organized by category, including their configuration parameters.

Detailed pages are available for more complex stages:

• ChunkText — Split long text fields into chunks for embedding and RAG pipelines.
• EmbeddedPython — Run Python code inside the JVM using GraalPy.
• ExternalPython — Delegate processing to an external Python process via Py4J.
• PromptOllama — Enrich documents using a locally-running LLM.
• QueryOpensearch — Execute OpenSearch search templates per document.

Plugin stages (TextExtractor, ApplyOCR, ApplyOpenNLPNameFinders, JlamaEmbed) are listed at the bottom of All Stages with their Maven dependencies.