Components on Lucille

Document

Thu, 23 Oct 2025 00:00:00 +0000

A Lucille Document is the basic unit of data that gets sent through a pipeline and indexed.

Documents are set of named fields which may contain single or list values represented in JSON. Each Document has a unique id.

Events

Thu, 23 Oct 2025 00:00:00 +0000

Lucille Events

As a document passes through various stages of the Lucille ETL pipeline, and as the documents are handled by the indexer, Event messages are generated.

Connectors listen for these events to ensure that all of the documents sent for processing are successfully processed and accounted for.

Errors during processing are reported and tracked via these event messages so the connector can report back the overall success or failure of the execution.

Publisher

Thu, 23 Oct 2025 00:00:00 +0000

Publisher provides a way to publish documents for processing by the pipeline. When published, a Lucille document becomes available for consumption by any active pipeline Worker.

The Publisher is aware of every document in the run that needs to be indexed, and determines when to end a run by reading all the specific document events.

Publisher also:

Is responsible for stamping a designated run_id on each published document and maintaining accounting details specific to a run.
Accepts incoming events related to published documents and their children (via in-memory queue or Kafka Event Topic).

Runner

Thu, 23 Oct 2025 00:00:00 +0000

When you run Lucille at the command line in standalone mode, you are invoking the Runner. When invoked, the runner reads the configuration file and then begins a Lucille run by launching the appropriate connector(s) and publisher. The runner generates a runId per Lucille run and terminates based on messages sent by the Publisher.

What the runner invokes can be thought of is an end to end Lucille Run.

A Lucille Run is a sequence of connectors to be run, one after the other. Each connectors feeds to a specific pipeline. A run can include multiple connectors feeding multiple pipelines.

Worker

Thu, 23 Oct 2025 00:00:00 +0000

File Handlers

Fri, 28 Feb 2025 00:00:00 +0000

File Handlers accept an InputStream for processing, and return the Documents they extract in an Iterator. The provided InputStream and any other underlying resources are closed when the Iterator returns false for hasNext(). As such, when working directly with these File Handlers, it is important to exhaust the Iterators they return.

File Handlers (Core)

CSV File Handler: Extracts documents from a csv file.
JSON File Handler: Extracts documents from a json (or a jsonl) file.
XML File Handler: Extracts documents from an xml file.

Custom File Handlers

Developers can implement and use custom File Handlers as needed. Extend BaseFileHandler to get started. To use a custom FileHandler, you have to reference its class in its Config. This is not needed when using the File Handlers provided by Lucille. You can override the File Handlers provided by Lucille, as well - just include the class you want to use in the Config.

Pipeline

Tue, 15 Oct 2024 00:00:00 +0000

A Pipeline refers to the complete sequence of stages that can be configured to transform documents with Lucille.

In most the most common architecture, a Pipeline is fed by a Connector and emits documents to an Indexer.