Components

Conceptual guide to the core components of Lucille and how they work together.

Lucille is built from a small set of components that work together to move data from source systems into a search backend.

Component	Role
Connectors	Read data from a source system and emit Documents into the pipeline.
Stages	Process and enrich Documents. Composed into Pipelines.
Indexers	Batch processed Documents and send them to the search backend.
Pipeline	An ordered sequence of Stages applied to each Document.
Worker	Pulls Documents from the source queue, runs them through the Pipeline, and forwards results.
Publisher	Tracks every Document from publication to terminal state; provides backpressure.
Runner	Orchestrates a complete run: validates config, starts components, waits for completion.
Document	The basic unit of data flowing through the system.
Config	Why Lucille is configuration-driven, and how HOCON and Typesafe Config shape the system.

For a catalogue of specific implementations — built-in connectors, stages, indexers, file handlers, and plugins — see the Ingest Design section.

The basic unit of data that is sent through a Pipeline and eventually indexed into a search engine.

A component that retrieves data from a source system and packages the data into Documents in preparation for transformation.

Provides a way to publish Documents for processing by the pipeline, and tracks their lifecycle until completion.

As Lucille runs, it generates Events that track document lifecycle and enable run completion accounting.

A Stage performs a specific transformation on a Document.

The ordered sequence of Stages that transform Documents before they are indexed.

A thread that retrieves published documents and passes them through a pipeline, then forwards completed documents to the Indexer.

An Indexer sends processed Documents to a specific destination.

Component that manages a Lucille Run end-to-end.

Why Lucille is configuration-driven, and how the choice of HOCON and Typesafe Config shapes the system.