Ingest Designer Guide on Lucille

Writing a Config

Mon, 09 Jun 2025 00:00:00 +0000

When you run Lucille, you provide a path to a configuration file that defines your entire ingest: which sources to read from, which pipelines to apply, and where to send the results. Configuration files use HOCON, a superset of JSON.

Quick references:

s3-opensearch.conf — A simple, runnable example that ingests files from S3 into OpenSearch.
application-example.conf — A comprehensive, annotated illustration of every top-level property and block that a Lucille config can contain (worker, publisher, kafka, indexer, etc.), excluding the implementation-specific parameters of individual stages, connectors, and indexers. Use this as a reference when you need to know what options exist and what they do. Note: this file is a reference document, not a tested runnable config.
lightbend/config — The upstream documentation for HOCON syntax and the Typesafe Config library that Lucille uses to parse configs. Consult this when you need the definitive rules on substitution syntax, include semantics, value concatenation, or other HOCON language features beyond what this guide covers.

Required Elements

A complete config file must contain three elements:

Defining Pipelines

Mon, 09 Jun 2025 00:00:00 +0000

Defining a Pipeline

Pipelines are defined in the pipelines list in your config file. Each pipeline requires a name and a stages list:

pipelines: [
 {
 name: "my-pipeline"
 stages: [
 {
 class: "com.kmwllc.lucille.stage.RenameFields"
 fieldMapping: { old_name: new_name }
 },
 {
 class: "com.kmwllc.lucille.stage.TrimWhitespace"
 fields: ["title", "description"]
 }
 ]
 }
]

Stages execute in the order they are listed. Each Stage receives the Document as mutated by all previous Stages.

Connecting a Connector to a Pipeline

Each Connector specifies which pipeline will process its output via the pipeline field:

Control Flow

Fri, 23 May 2025 00:00:00 +0000

This page is relevant to both ingest designers (working in config only) and component developers (writing Java stage and connector code). Each section identifies which approach applies to you.

Conditional Stage Execution

Config | When to use: You want a stage to run only on documents that have a particular field, or a particular value in a field. For example, only run an embedding stage on documents where content_type is "article", or only run a cleanup stage on documents that have a raw_html field.