Lucille is a production-grade Search ETL framework for loading data into Lucene-based search engines — Apache Solr, Elasticsearch, and OpenSearch — and vector databases such as Pinecone and Weaviate. It supports batch, incremental, and streaming ingestion with inline document enrichment, and runs as a single JAR with no persistent server to manage. Lucille is Java-based and open-source, developed and maintained by KMW Technology.

How Lucille Works

A Connector retrieves data from a source system and publishes it as Lucille Documents.
Workers route each Document through a configurable enrichment Pipeline built from composable Stages.
An Indexer sends the processed Documents to the search backend.
These three components communicate through a messaging layer — either in-memory queues (local mode, single JVM) or Apache Kafka (distributed mode, multiple JVMs at scale).
Document lifecycle events — publication, processing, indexing, failure, and drops — are tracked so Lucille knows when a run is complete and can report exact success and failure counts.

For a deeper look at the architecture, see Architecture Overview.

1.1 - Why Lucille

What Lucille does, what it excels at, and how to know if it’s right for your problem.

What Lucille Does

Lucille is a Java-based framework for getting data into search engines and vector databases. It reads from source systems (databases, filesystems, APIs, Kafka topics), enriches the data through a configurable pipeline of processing stages, and delivers it to a search backend (Solr, OpenSearch, Elasticsearch, Pinecone, Weaviate) in batches.

The framework handles the hard parts of production search ingestion — concurrency, completion detection, fault tolerance, batching, retry, error handling, backpressure, and testability — so that you can focus on the specifics of your data and your enrichment logic.

What Lucille Excels At

Search-native indexing. Lucille’s indexers understand search engine bulk APIs, batching semantics, deletion markers, index routing, and retry logic. The document model represents data the way search engines think about it — typed fields, single-valued and multi-valued, with operations that map directly to search engine semantics. You don’t adapt a generic ETL tool to speak search; the framework already does.

Inline enrichment. A large library of built-in stages covers common search ingestion tasks without custom code: text extraction, NLP, entity recognition, language detection, embedding generation, text chunking, database lookups, HTTP enrichment, scripting (JavaScript, Python), and JSONata transformations. Enrichment happens inside the pipeline as documents flow through — not as a separate post-processing step.

Scalability without code changes. The same pipeline configuration runs in a single JVM for development and in a fully distributed Kafka deployment for production. Scaling up means adding Worker threads (local mode) or Worker processes (distributed mode). The transition is a command-line flag, not a rewrite.

Operational simplicity. Lucille is a JAR you run from the command line. It starts, processes documents, and exits. There is no server to keep running between jobs, no management UI to maintain, and no deployment infrastructure beyond the JVM.

Exact document accounting. Every document is tracked from publication through its terminal state. At the end of a run, you know exactly how many documents succeeded, failed, or were dropped — not an approximation.

Fault tolerance in distributed mode. Crash resilience via Kafka consumer group rebalancing, poison pill detection with dead-letter queues, and graceful shutdown that preserves in-flight work. One bad document doesn’t stop the run; one crashed Worker doesn’t lose data.

Configuration-driven operation. All aspects of an ingest — sources, pipelines, stages, destinations, tuning parameters — are declared in a readable HOCON configuration file. Changes are possible by editing the file alone, without rebuilding the project. Configuration is composable, parameterizable, and validatable before execution.

Testability. The entire framework runs inside a unit test with no external infrastructure. A dedicated test mode captures the complete history of every document for assertion, making it straightforward to verify pipeline correctness.

When Lucille Is the Right Choice

Lucille is a strong fit when:

Your destination is a search engine or vector database — Solr, OpenSearch, Elasticsearch, Pinecone, or Weaviate.
Your pipeline involves non-trivial enrichment — NLP, embeddings, OCR, database lookups, LLM-based extraction, text chunking — not just field mapping.
You need both batch and streaming from the same pipeline — start with a batch backfill, transition to continuous streaming updates, without rewriting anything. Batch mode gives you exact accounting and structured run summaries; streaming mode gives you unbounded continuous ingestion. The same pipeline logic works in both.
You want to start simple and scale later — develop in local mode, deploy distributed when volume demands it, without changing pipeline logic.
You prefer code-and-config over UI-driven workflows — pipelines are version-controlled HOCON files, not visual diagrams.
You’re a Java shop (or willing to use Java for custom components) — the framework and extension points are Java-native.

How to Know If It’s Not the Right Fit

Lucille is probably not the best choice when:

Your destination is a data warehouse (Snowflake, BigQuery, Redshift). Lucille’s indexers target search engines and vector databases.
You need a massive SaaS connector catalog. Lucille’s built-in connectors cover databases, filesystems, Kafka, RSS, and search engines. If you need to ingest from hundreds of SaaS applications (Salesforce, HubSpot, Zendesk), tools with larger connector ecosystems may be more practical.
Sub-second real-time indexing latency is a hard requirement. Lucille’s streaming mode is continuous but routes through Kafka, a Worker pipeline, and a batching Indexer — adding latency that dedicated CDC-to-search connectors can avoid.
A visual workflow editor is required. Lucille pipelines are defined in configuration files. There is no drag-and-drop UI or graphical pipeline designer.
Your team has no Java engineers and no interest in Java. Custom connectors and stages are Java classes. The EmbeddedPython and ExternalPython stages provide a Python authoring path for enrichment logic, but the framework itself is Java.

Other Tools to Consider

If Lucille isn’t the right fit, or if you want to evaluate alternatives, tools that teams commonly consider for search-adjacent ETL include:

Logstash — input/filter/output pipeline model, tightly integrated with the Elastic ecosystem
OpenSearch Data Prepper — data collector oriented toward observability and log ingestion into OpenSearch
Apache NiFi — visual dataflow tool with a large processor library and a persistent server model
Apache Spark — distributed data processing framework suited to petabyte-scale workloads
Airbyte — replication tool with a large SaaS connector catalog, oriented toward data warehouses
Custom pipelines — Python scripts, Java applications, or Spark jobs grown organically over time

Each has different strengths and tradeoffs. Lucille’s niche is the intersection of search-native indexing, inline enrichment, and operational simplicity — a purpose-built tool for the specific problem of getting enriched data into search engines at scale.

1.2 - Installation

How to get Lucille — as a Maven dependency, from a source build, or for development.

Do You Need to Install Lucille?

It depends on how you’re using it:

Using Lucille as a Library in an Existing Java Project

If you’re embedding Lucille inside an existing Java project (e.g., calling Runner.run() programmatically, or using Lucille’s Document API and Stages within your own application), you don’t need to install anything. Just add Lucille as a Maven dependency:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.kmwllc</groupId>
      <artifactId>lucille-bom</artifactId>
      <version>0.9.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.kmwllc</groupId>
    <artifactId>lucille-core</artifactId>
  </dependency>
  <!-- Add plugins as needed -->
  <dependency>
    <groupId>com.kmwllc</groupId>
    <artifactId>lucille-tika</artifactId>
  </dependency>
</dependencies>

Lucille is published to Maven Central. The BOM ensures all Lucille modules are at the same version. No source build required.

Running Lucille from the Command Line (Runner, Worker, Indexer)

If you’re launching Lucille components via java -cp ... com.kmwllc.lucille.core.Runner (or Worker, WorkerIndexer, Indexer), you need the compiled JARs and their dependencies on the classpath. The easiest way to get these is to clone the repository and build from source:

git clone https://github.com/kmwtechnology/lucille.git
cd lucille
git checkout v0.9.0  # check out the release tag/branch you want to use
mvn clean install

After the build, the JARs and dependencies are in each module’s target/ directory. You can then run Lucille with:

java -Dconfig.file=my-config.conf -cp 'lucille-core/target/lucille-core-0.9.0.jar:lucille-core/target/lib/*' com.kmwllc.lucille.core.Runner

Developing New Lucille Components (Stages, Connectors, Indexers)

If you’re contributing to Lucille or developing new components that will be part of the Lucille codebase, clone the repository and build from source as described below.

Prerequisites

To build and run Lucille from source, you need:

Java 17+ JDK (not just a JRE)
Maven (recent version)

Java Setup (JDK 17+ Required)

Important: Before running any Lucille commands, make sure JAVA_HOME points to a JDK 17+ (not just a JRE) and that $JAVA_HOME/bin is on your PATH (or %JAVA_HOME%\bin on Windows). Maven and the java launcher rely on this.

Verify Java

java -version

You should see version 17 (or newer). If it’s missing or older than 17 install a JDK 17+ using one of the options below.

Install Options

Package manager

macOS (Homebrew)
```
brew install openjdk@17
```
Windows (Chocolatey)
```
choco install microsoft-openjdk17
```

Vendor installer

Download a JDK 17+ installer from a vendor such as Oracle JDK.
Run the installer, then set JAVA_HOME as shown below.

Set `JAVA_HOME` and `PATH`

macOS

export JAVA_HOME="$(/usr/libexec/java_home -v 17)"
export PATH="$JAVA_HOME/bin:$PATH"

Windows

Open System Properties, Environment Variables.
Create/Edit JAVA_HOME and point it to your JDK folder.
Edit Path and add %JAVA_HOME%\bin above other Java entries.

Maven Setup

mvn -v

You should see a recent Maven version and your Java home. If mvn is not found, install Maven using one of the options below.

Install Options

Package manager

macOS (Homebrew)
```
brew install maven
```
Windows (Chocolatey)
```
choco install maven
```

Binary installer

Download the binary zip/tar for Apache Maven from the official website.
Add Maven’s bin/ to your PATH.

macOS

export PATH="<maven-dir>/bin:$PATH"

Windows

Open System Properties, Environment Variables.
Edit Path and add <maven-dir>/bin.

Clone the Repository

git clone https://github.com/kmwtechnology/lucille.git
cd lucille

To use a specific release version, check out the corresponding tag:

git checkout 0.9.0  # replace with the version you want

To see available release tags:

git tag --list

Build Lucille

mvn clean install

This compiles all modules and produces build artifacts under each module’s target/ folder. After the build:

lucille-core/target/lucille-core-{version}.jar — the core framework JAR
lucille-core/target/lib/ — all runtime dependencies
lucille-plugins/lucille-tika/target/lucille-tika-{version}.jar — plugin JARs (one per plugin)
lucille-examples/*/target/lib/ — example projects with all dependencies copied for easy classpath setup

1.3 - Your First Pipeline

A step-by-step walkthrough of writing a minimal Lucille config from scratch.

This page walks you through creating a Lucille configuration file from scratch — a single connector reading from a CSV file, a pipeline with two stages, and an indexer writing to a local CSV output. No search backend required.

By the end, you’ll understand the structure of a Lucille config and be ready to adapt it for your own data.

Prerequisites

Lucille built from source (see Installation)
A terminal in the Lucille repository root

Step 1: Create a Source File

Create a file called my-source.csv with a small set of classic novels:

id,title,author,year
1,The Great Gatsby,F. Scott Fitzgerald,1925
2,To Kill a Mockingbird,Harper Lee,1960
3,1984,George Orwell,1949
4,Pride and Prejudice,Jane Austen,1813
5,The Catcher in the Rye,J.D. Salinger,1951

Step 2: Write the Config File

This config reads my-source.csv, applies two field transformations to each row, and writes the results to a new CSV file. This config has three parts:

Connector — a FileConnector with the CSV file handler reads my-source.csv and emits each row as a Document.
Pipeline — two stages manipulate fields on each Document: one renames title to book_title, and one adds a static source tag to every Document.
Indexer — a CSV indexer writes the transformed Documents to an output file. No search backend is needed for this first example.

Create a file called my-first-pipeline.conf:

# ─── CONNECTORS ───────────────────────────────────────────────
# A list of connectors to run in sequence.
# Each connector reads from a source and publishes Documents.

connectors: [
  {
    # A name for this connector (appears in logs and run summary)
    name: "csv-reader"

    # The connector implementation class
    class: "com.kmwllc.lucille.connector.FileConnector"

    # Which pipeline should process this connector's documents
    pipeline: "my-pipeline"

    # Path(s) to the source file(s)
    paths: ["my-source.csv"]

    # Apply the CSV file handler (with default options) to *.csv files.
    # Each CSV file is parsed row by row; one Document is emitted per row.
    fileHandlers: {
      csv: {}
    }
  }
]

# ─── PIPELINES ────────────────────────────────────────────────
# Each pipeline is a named sequence of Stages.
# Stages transform Documents in place before they reach the Indexer.

pipelines: [
  {
    name: "my-pipeline"

    stages: [
      # Stage 1: Rename "title" to "book_title"
      {
        name: "rename-title"
        class: "com.kmwllc.lucille.stage.RenameFields"
        fieldMapping: {
          "title": "book_title"
        }
      },

      # Stage 2: Add a static field to every document
      {
        name: "add-source-tag"
        class: "com.kmwllc.lucille.stage.SetStaticFieldValue"
        fieldName: "source"
        fieldValue: "my-first-pipeline"
      }
    ]
  }
]

# ─── INDEXER ──────────────────────────────────────────────────
# The Indexer sends processed Documents to a destination.
# We use the CSV indexer here so no search backend is needed.

indexer {
  type: "CSV"
}

# CSV indexer settings
csv {
  # Which fields to include in the output (must match field names after pipeline processing)
  columns: ["id", "book_title", "author", "year", "source"]

  # Where to write the output
  path: "my-output.csv"

  # Include a header row
  includeHeader: true
}

Step 3: Run It

java -Dconfig.file=my-first-pipeline.conf \
  -cp 'lucille-core/target/lucille-core-*.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner

Step 4: Check the Output

Open my-output.csv:

id,book_title,author,year,source
1,The Great Gatsby,F. Scott Fitzgerald,1925,my-first-pipeline
2,To Kill a Mockingbird,Harper Lee,1960,my-first-pipeline
3,1984,George Orwell,1949,my-first-pipeline
4,Pride and Prejudice,Jane Austen,1813,my-first-pipeline
5,The Catcher in the Rye,J.D. Salinger,1951,my-first-pipeline

The title field was renamed to book_title, and a source field was added to every document.

Step 5: Check the Logs

You should see output like:

INFO  Runner: Pipeline Configuration is valid.
INFO  Runner: Connector Configuration is valid.
INFO  Runner: Indexer Configuration is valid.
INFO  Runner: Starting run with id ...
INFO  Runner: Running connector csv-reader feeding to pipeline my-pipeline
INFO  PublisherImpl: First doc published after ... ms
INFO  WorkerPool: 5 docs processed. One minute rate: ... docs/sec. Mean pipeline latency: ... ms/doc.
INFO  Stage: Stage rename-title metrics. Docs processed: 5. Mean latency: ... ms/doc. Children: 0. Errors: 0.
INFO  Stage: Stage add-source-tag metrics. Docs processed: 5. Mean latency: ... ms/doc. Children: 0. Errors: 0.
INFO  Runner: RUN SUMMARY: Success. 1/1 connectors complete. All published docs succeeded.
INFO  Runner: Run took ... secs.

Understanding the Config Structure

Every Lucille config has three required parts:

Block	Purpose
`connectors`	A list of data sources to read from, executed in sequence
`pipelines`	Named sequences of Stages that transform Documents
`indexer`	Where to send the processed Documents

Each connector specifies:

name — for logging and the run summary
class — the fully qualified Java class that implements the connector
pipeline — which pipeline processes this connector’s output
Connector-specific settings (e.g., paths for FileConnector)

Each stage in a pipeline specifies:

class — the fully qualified Java class that implements the stage
name (optional) — for logging and per-stage metrics
Stage-specific parameters (e.g., fieldMapping for RenameFields)

The indexer specifies:

type — a shorthand for built-in indexers (Solr, OpenSearch, Elasticsearch, CSV)
Or class — for plugin or custom indexers
Backend-specific settings in a separate block (e.g., csv {}, solr {}, opensearch {})

What Have We Accomplished?

At first glance, the config above might feel disproportionate to the task. We renamed one field and added a static value — a transformation you could express in a single line of shell:

sed '1s/title/book_title/' my-source.csv \
  | awk -F, 'BEGIN{OFS=","} NR==1{print $0,"source"} NR>1{print $0,"my-first-pipeline"}' \
  > my-output.csv

That’s fair. For this specific transformation on this specific file, a shell one-liner wins. The value of Lucille comes from what you can do next — and how little you have to change to get there.

1. Index to any search backend

Replace the CSV indexer with a real search backend — OpenSearch, Elasticsearch, or Solr — and your trivial file experiment becomes a real search ingestion pipeline. For OpenSearch:

indexer {
  type: "OpenSearch"
}

opensearch {
  url: ${?OPENSEARCH_URL}
  index: "my-books"
}

That’s the only change. The connector, the pipeline, the stages — everything else stays exactly the same. If you later want to switch from OpenSearch to Solr, you change the indexer block and add a solr {} block. No code, no build cycle, no rewrite.

2. Iterate on your pipeline

Add stages, chain transformations, and introduce conditions — all in config. Want to generate an embedding for each book title and store it as a vector field? Add a stage. Want to skip that stage for documents that already have an embedding? Add a condition. Browse the Stage Reference for the full library of available transformations.

None of this requires a build. Edit the config file, rerun Lucille, see the results. The pipeline is the config.

3. Scale

In our example, five rows processed in milliseconds and scale was not a concern. But now imagine the CSV has a million rows, each containing an S3 URL to a large PDF. Your pipeline must fetch each PDF, extract text with Tika, run OCR on scanned pages, and generate embeddings. That pipeline takes hours or days on a single thread.

Step 1: add worker threads. Lucille’s local mode uses a thread pool for document processing. Increase the number of threads to parallelize across available CPU cores and overlap IO:

worker {
  threads: 8
}

No other changes needed. The same config, more threads.

Step 2: distribute across machines. If one machine isn’t enough, switch to distributed mode. Lucille’s components — Runner, Workers, and Indexer — run as separate JVM processes and communicate through Kafka. Here’s what changes:

The Runner gets the -usekafka flag: com.kmwllc.lucille.core.Runner -usekafka
Workers and the Indexer are started as separate processes on separate machines (or in separate Kubernetes pods): com.kmwllc.lucille.core.Worker my-pipeline and com.kmwllc.lucille.core.Indexer my-pipeline
A kafka {} block with bootstrapServers is added to the config

Your connector definition and your pipeline stages are untouched. See Production Deployment for the full distributed mode setup.

At first, with a trivial five-row example that ran in under a second, Lucille looked like overkill compared to a shell script. But as soon as we wanted to ingest into a real search backend, iterate on enrichment logic, and scale out to handle large volumes of heavy documents, we could do all of that in a config-driven way — switching backends, adding stages, and distributing across machines — without having to solve any of those problems from scratch.

Consider what Lucille is doing on our behalf in the scaled-out version of this pipeline. The Runner is reading our million-row CSV and producing each row as a Document onto a Kafka topic, applying backpressure so it doesn’t publish faster than Workers can consume. Multiple Worker processes — running on different machines or in different Kubernetes pods — are each pulling Documents off that topic in parallel, each running multiple threads, each thread independently fetching a PDF from S3, extracting text, running OCR, and generating embeddings. As each Document finishes processing, a lifecycle event is sent back to the Runner so it can track exactly how many Documents have succeeded, failed, or been dropped. If a Worker crashes mid-run, Kafka’s consumer group protocol detects it and reassigns its unacknowledged Documents to another Worker — nothing is silently lost. If a single malformed PDF repeatedly crashes a Worker, Lucille routes it to a dead-letter topic after a configurable number of retries and keeps the rest of the run going. The Indexer is collecting processed Documents and sending them to the search backend in tunable batches, managing bulk API semantics, and handling retries on transient backend errors. Throughout the run, Lucille is logging throughput metrics, per-stage latency, and document counts at regular intervals. When the last Document reaches a terminal state, Lucille determines that the run is complete and prints a structured summary — how many succeeded, how many failed, how many were dropped, and how long it took.

All of that happens without any code on our part. Our contribution is the config file we wrote in Step 2.

1.4 - Running Lucille

How to run Lucille in local and distributed mode, with examples and guidance on verifying a run.

Try the Examples

Lucille’s lucille-examples module contains several runnable examples:

Example	Description
lucille-simple-csv-solr-example	Ingest a CSV file into Solr in local mode. A good first example.
lucille-distributed-example	Full distributed mode with Runner, Worker, Indexer, Kafka, ZooKeeper, and Solr each running in their own Docker container via Docker Compose. Includes an integration test that verifies the ingest result. Run with `mvn verify -Pnightly` from that directory.

To run the simple CSV example, start an instance of Apache Solr on port 8983 and create a collection called quickstart. Then from lucille-examples/lucille-simple-csv-solr-example, run:

mvn clean install
./scripts/run_ingest.sh

This executes Lucille with simple-csv-solr-example.conf, which reads a CSV of top songs and sends each row as a document to Solr. After the run, issue a commit with openSearcher=true on your quickstart collection and run a *:* query in the Solr admin dashboard to see the indexed documents.

Local Mode

Local mode runs all Lucille components — Connector, Workers, and Indexer — as threads inside a single JVM. No Kafka or external messaging is required. This is the right mode for development, testing, and single-machine production runs.

Prepare a Configuration File

Point Lucille at a config file declaring your connectors, pipelines, and indexer. See Configuration Management for the full schema, or follow Your First Pipeline for a step-by-step walkthrough.

Run the Runner

From the repository root:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner

Flag	What it does
`-Dconfig.file=...`	Path to your HOCON configuration file
`-cp '...'`	Lucille core JAR and all runtime dependencies
`com.kmwllc.lucille.core.Runner`	Boots the Lucille engine in local mode and runs the configured pipeline to completion

See the troubleshooting guide if Lucille doesn’t start as expected.

Distributed Mode

Distributed mode runs each Lucille component as its own JVM process, communicating through Apache Kafka. Use this when you need to scale out across multiple machines or processes.

This guide assumes Kafka and your destination system are already running and reachable. For production deployment patterns including Docker Compose and Kubernetes, see Production Deployment.

Prepare a Configuration File

Use a single config file shared by all components. It must declare your connector(s), pipeline(s), kafka configuration, and your indexer with its backend config.

Start the Runner

The Runner publishes documents to the Kafka source topic, listens for pipeline run events, logs run statistics, and waits for the run to complete:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner \
  -usekafka

Start Workers

Each Worker consumes documents from the Kafka source topic, processes each document through the configured pipeline, and writes processed documents to the Kafka destination topic:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Worker \
  <pipeline-name>

Start Workers before the Runner to avoid Kafka consumer group rebalancing delays while documents are already in flight.

Start the Indexer

The Indexer consumes documents from the Kafka destination topic and sends batches to the configured search backend:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Indexer \
  <pipeline-name>

Flag	What it does
`-Dconfig.file=...`	Path to your HOCON configuration file
`-cp '...'`	Lucille core JAR and all runtime dependencies
`Runner -usekafka`	Starts the run and coordinates with Kafka
`Worker <pipeline-name>`	Processes documents through the named pipeline
`Indexer <pipeline-name>`	Writes processed documents to the configured backend

See the troubleshooting guide if components don’t connect as expected.

Verifying Your Run

During the Run

Lucille logs throughput and latency metrics periodically:

25/10/31 13:40:21 INFO WorkerPool: 27017 docs processed. One minute rate: 1787.10 docs/sec. Mean pipeline latency: 10.63 ms/doc.
25/10/31 13:40:22 INFO PublisherImpl: 37029 docs published. One minute rate: 3225.69 docs/sec. Waiting on 21014 docs.
25/10/31 13:40:22 INFO Indexer: 17016 docs indexed. One minute rate: 455.07 docs/sec. Mean backend latency: 6.90 ms/doc.

At Completion

At the end of every run, Lucille prints a stage-by-stage performance summary followed by a final run result:

25/10/31 13:46:47 INFO Stage: Stage rename-title metrics. Docs processed: 200000. Mean latency: 0.0003 ms/doc. Children: 0. Errors: 0.
25/10/31 13:46:47 INFO Stage: Stage add-source-tag metrics. Docs processed: 200000. Mean latency: 0.3532 ms/doc. Children: 0. Errors: 0.
25/10/31 13:46:47 INFO Runner:
RUN SUMMARY: Success. 1/1 connectors complete. All published docs succeeded.
connector1: complete. 200000 docs succeeded. 0 docs failed. 0 docs dropped. Time: 416.47 secs.
25/10/31 13:46:47 INFO Runner: Run took 417.46 secs.

Check Your Search Backend

After the run, verify that documents are visible in your target system. For Solr, a commit with openSearcher=true is required before documents appear in query results — Lucille does not issue a commit automatically. For Elasticsearch and OpenSearch, documents are available after the index refresh interval (default: 1 second).

2 - Architecture

Understanding Lucille’s core components, topology, and design.

Lucille separates the three concerns of ETL — reading, transforming, and writing — into distinct components that run concurrently.

Start Here

Section	What It Covers
Overview	The problem Lucille solves, the core architecture, topology, document lifecycle, and design rationale
Components	Reference pages for each component (Document, Pipeline, Publisher, Connectors, Indexers, Stages, Config)

How Components Interact

The components communicate through queues. In local mode these are in-memory LinkedBlockingQueue instances. In distributed mode they are Kafka topics. The component code is identical in both cases — only the messenger implementation changes.

Connector → [processing queue] → Worker(s) → [indexing queue] → Indexer
                                       ↓ events ↑
                                  Publisher (run accounting)

For in-depth explanations of how each subsystem works internally, see Internals.

2.1 - Overview

A narrative introduction to Lucille’s architecture — the problem it solves, the core design, and how documents flow through the system.

Read these pages in order for a complete introduction to Lucille’s architectural design.

The Problem of Search ETL — Why a simple sequential ingest loop falls short in production and what needs to change.
Parallelizing Search ETL — How Lucille maps the three ETL functions to concurrent components communicating through queues, and how it tracks document lifecycle across an asynchronous system.
From Single Process to Distributed — How the same pipeline code runs identically in a single JVM or distributed across machines, and how the Messenger abstraction makes this possible.
Topology — Batch and streaming models, the WorkerIndexer, and deployment configurations from single-JVM to fully distributed.
Document Lifecycle — The complete journey of a single Document through the system, plus cross-cutting concerns like logging, metrics, and testing.
Design Rationale — The 24 requirements that govern Lucille’s architecture and why the system is designed the way it is.

2.1.1 - The Problem of Search ETL

Why a simple sequential loop falls short for production search ingestion, and how Lucille addresses the pitfalls.

Organizations build search applications when their users need to find information across large, heterogeneous collections of data — product catalogs, legal documents, support tickets, research papers, internal wikis, file shares. The data lives in databases, cloud storage, APIs, and content management systems, often in different formats and with varying levels of structure. Before any of it becomes searchable, it must be extracted from these sources, cleaned, enriched (with metadata, classifications, embeddings, or extracted entities), and delivered to a search engine or vector database in the right format. This process — getting source data into a search-ready state — is search ETL.

Getting data into a search engine involves three distinct functions:

Connect to a source system to acquire data.
Clean and enrich the data.
Send the data to the search backend.

In a first prototype of a search ingestion process, it is common to perform these functions in a sequential loop:

// as long as the source system has more data for us to consume…
while (source.hasNext()) {
    // 1 – create a Document from the next record
    Document doc = Document.from(source.next());
    // 2 – pass the Document through an enrichment pipeline
    pipeline.process(doc);
    // 3 – send the document to the search engine
    searchClient.index(doc);
}

When you’re ingesting a few hundred documents in a POC, this loop finishes in seconds. It works. It’s easy to reason about. At this point, adopting a search ETL framework feels like overkill.

Then you go to production.

The dataset is no longer a few hundred documents — it’s millions. The loop that finished in ten seconds now takes a full day. And over the course of that day, you run into a series of problems that the simple loop has no answer for.

1. One bad document stops everything

Somewhere in that day-long run, a document fails. Maybe the enrichment logic throws an exception on an unexpected input. Maybe the search backend rejects a malformed field. The loop stops.

Do you fix the document and restart from the beginning? That means re-ingesting everything you already indexed. Do you build checkpointing logic so you can resume from where you left off? Now you need a persistent state store. Or do you skip the bad document and keep going? If you do, where does the failed document go? How do you know at the end of the run which documents succeeded and which didn’t? How do you inspect and reprocess just the failures?

The simple loop gives you no framework for any of these decisions. And search backends make this harder than general ETL in a specific way: bulk indexing APIs return mixed results. A single bulk request can partially succeed — some documents accepted, others individually rejected due to a field type mismatch or a mapping conflict — and the response requires per-document inspection to know what actually happened. You can’t treat indexing as a simple pass/fail operation and retry the batch whole. You need per-document error tracking inside every batch.

2. You can’t see what’s happening

The run has been going for six hours. Is it on track? Is it stuck? Are there bottlenecks building up somewhere? How many documents have been indexed so far, and how many are still waiting?

The loop has no instrumentation. You have no visibility into throughput, no per-stage latency breakdown, no count of in-flight documents, no way to distinguish “slow but progressing” from “silently stuck.” You’re watching a process run for a day with no dashboard and no progress bar.

3. Enrichment logic becomes tangled and unreusable

As the pipeline grows — adding text extraction, NLP, embedding generation, database lookups — the enrichment logic accumulates inside or alongside the loop. It becomes difficult to adjust individual steps without touching the whole thing. Testing a change requires running the full ingest or building a separate harness.

When a second project needs some of the same transformations, there’s no clean mechanism for reuse. The logic gets copied or extracted into a shared library that gradually accumulates its own complexity. Configuration for which steps run and in what order is hardcoded rather than declared.

4. One source document fans out into many index records

In general ETL, a record in the source maps to a record in the destination. Search ingestion breaks that assumption.

For semantic search and RAG pipelines, a single source document — a PDF, a support ticket, a product page — often needs to be split into smaller chunks, each indexed as an independent search record so that retrieval can return the most relevant passage rather than the entire document. One source document becomes tens or hundreds of index records.

The simple loop has no model for this. searchClient.index(doc) indexes one document. If a stage in the pipeline splits a document into chunks, those chunks need to flow through the rest of the pipeline independently, be tracked individually for accounting purposes, and be indexed as separate records. The parent document and its chunks may also need to be indexed in a specific order if the chunks reference the parent by ID.

This 1-to-N fan-out is unique to search ingestion. It doesn’t arise in database ETL, and it breaks both the loop’s execution model and any completion-tracking logic built on top of it.

5. Environment-specific settings are hardcoded

The pipeline runs against a development search backend today. Next week it needs to run against staging, with different credentials, a different Solr URL, and a different collection name. After that, production — with its own connection details, possibly different Kafka brokers, and secrets that shouldn’t be hardcoded.

Managing that across environments — without duplicating code, without accidentally running dev config against prod, with a clear way to inject secrets at runtime — is not a problem the loop was designed to solve.

6. The sequential loop can’t go faster

This is the hardest problem, and the reason all the others matter more once you solve it.

If a full dataset ingest takes a day, the obvious question is: can we make it faster? The answer is yes — but not by iterating on the sequential loop. To go faster, you have to parallelize.

Part of what makes this problem harder in search ingestion than in general ETL is the cost of the enrichment step itself. In database-to-database ETL, transformation is often lightweight — type conversion, field mapping, simple lookups. In search ingestion, enrichment exists specifically to pre-compute things that make search fast and relevant: OCR on scanned documents, named entity extraction, classification, vector embedding generation. This work is genuinely expensive per document. The pipeline is CPU-bound in a way that most ETL pipelines are not, which is exactly why parallelizing it matters so much — and why the gains from doing it correctly are so large.

The loop is slow because its three steps have fundamentally different performance characteristics: connecting to the source system is I/O-bound, pipeline enrichment is often CPU-bound (model inference, API calls, computation), and indexing is I/O-bound but also depends on the CPU of the search backend. The sequential loop adds the latency of all three steps per document:

total time = ((source latency) + (pipeline latency) + (indexing latency)) × (number of documents)

To do better, you need to run these three phases concurrently — reading the next batch of documents while the previous batch is being enriched, while the batch before that is being indexed. And you need multiple enrichment workers running in parallel to saturate available CPU.

But parallelizing the loop creates its own roadblocks. Multiple reader threads need to coordinate so they don’t fetch the same records twice. The enrichment stage may hold expensive resources — models, connection pools — that can’t safely be shared across threads. The search backend performs best when documents arrive in batches, not one at a time, which requires accumulating results from multiple parallel workers before flushing. And if you scale to multiple machines, you need a message queue to distribute work across processes, which means managing Kafka topics, consumer groups, and partition assignment.

Then, once you succeed in parallelizing, a new set of problems emerges. How do you know when the run is complete? With a sequential loop, the answer is obvious: the loop returns. With parallel workers, you can’t know the run is done just because the source is exhausted — documents are still in flight, being processed and indexed. You need an accounting system that tracks every document from publication through its terminal state. How do you handle backpressure? A fast source feeding slow enrichment workers will produce documents faster than they can be consumed, eventually exhausting memory. How do you handle a worker crash mid-run? Kafka’s consumer group protocol can reassign its partitions, but unacknowledged documents need to be redelivered without silently dropping or double-indexing them. What about a document that repeatedly crashes a worker — a malformed input that triggers a bug in third-party code? Without a retry limit and a dead-letter queue, one bad document can loop forever.

Lucille is the system you arrive at after encountering these problems in real production deployments, solving them one by one, and validating the solutions at scale. It is not a theoretical framework — it is the accumulated result of decisions forced by real failure modes.

2.1.2 - Parallelizing Search ETL

How Lucille structures search ingestion as three concurrent components communicating through queues, and how it tracks document lifecycle across an asynchronous system.

Lucille addresses all of the pain points described in The Problem of Search ETL — error handling, observability, pipeline composition, fan-out, configuration management, and scalability. This page focuses on the last and most fundamental of those problems: the sequential loop can’t go faster.

Lucille’s answer is a parallel architecture whose components derive directly from the structure of the problem. Each of the three functions of search ETL — acquiring data, enriching it, and indexing it — becomes its own independent concurrent component, operating at its own pace, limited only by its own resources rather than by the speed of the others.

To design any concurrent system, two questions must be answered:

What are the core components?
How do they communicate?

The Core Components

Connectors connect to source systems to acquire data. A Connector reads from its source — a filesystem, a database, a Kafka topic, an RSS feed — and emits Documents into the system one at a time. It does not know how many Workers will process those documents, or how long enrichment will take.

Workers enrich documents by passing them through a Pipeline of Stages. Each Stage performs a specific transformation: extracting text, running NLP, generating embeddings, looking up database records. A Worker handles one document at a time and writes the result onward. Multiple Workers run concurrently, each with its own Pipeline instance, so enrichment scales with available CPU.

Indexers receive processed documents and send them in batches to the search backend. An Indexer accumulates documents until a batch is full or a timeout expires, then issues a single bulk API call. Batching is essential for search engine performance — bulk writes are significantly faster than one-at-a-time indexing.

No component waits for another unless it exhausts its backlog of work — or the downstream queue is full, which provides natural backpressure on the upstream component.

How Should They Communicate?

In some distributed systems, asynchronous components require tight, fine-grained communication. Consider a distributed query engine where a central coordinator assigns specific data partitions to worker nodes, monitors their progress with heartbeats, detects straggler tasks, re-assigns failed partitions to other nodes, and waits for explicit acknowledgment from every worker before declaring a query complete. The coordinator and workers exchange a continuous stream of control messages — assignment, acknowledgment, status updates, failure notifications — and the coordinator must maintain a detailed model of the state of every worker at all times.

Lucille takes a different approach. The key observation is that Connectors, Workers, and Indexers can be largely decoupled. No component needs to know what the others are doing in real time. A Connector doesn’t need to know how many Workers exist or whether they are fast or slow — it just needs somewhere to put its output. A Worker doesn’t need to know which Connector produced a document or which Indexer will consume it. An Indexer doesn’t need to know where documents came from or how many are still in the pipeline behind them.

This means it suffices for the components to communicate via queues. Each component places its output on a queue for the next component to read from:

The Connector puts documents on a processing queue.
Workers read the processing queue, process documents through the Pipeline, and put results on an indexing queue.
Indexers read the indexing queue and send documents to the search backend.

No component calls another directly. No component waits for a response. Each simply produces to a queue and consumes from a queue. The queue is the entire communication contract.

// Shared queues connecting the three components.
Queue processingQueue;   // documents waiting to be enriched
Queue indexingQueue;     // enriched documents waiting to be indexed


// === CONNECTOR ===
while (source.hasNext()) {
    Document doc = Document.from(source.next());
    processingQueue.put(doc);
}


// === WORKER (N instances, running concurrently) ===
while (running) {
    Document doc = processingQueue.poll();
    Document result = pipeline.process(doc);
    indexingQueue.put(result);
}


// === INDEXER (M instances, running concurrently) ===
while (running) {
    Document doc = indexingQueue.poll();
    batch.add(doc);
    if (batch.isFull() || batch.isExpired()) {
        searchClient.bulkIndex(batch.flush());
    }
}

This is the core of Lucille’s architecture. The Connector, Workers, and Indexers need no knowledge of each other beyond the existence of these two queues.

What’s Missing?

The pseudocode above captures the essential structure but leaves out something important: there is no way to know when a batch ingest is complete, no way to track how many documents have succeeded or failed, and no support for two common pipeline operations — dropping documents that should not be indexed, and emitting child documents from a single source record.

With the sequential loop, completion was trivial:

while (source.hasNext()) { }
// loop returns → run is done

In a concurrent system, the Connector finishing is not enough. When the Connector stops putting documents on the processing queue, Workers may still be enriching documents, and Indexers may still be sending batches. The system is not done until every document has reached a terminal state — indexed, failed, or dropped.

Tracking this requires knowing about every document in flight: which ones have been introduced, which have completed, and which are still being processed. A Worker can also generate new child documents during pipeline processing — for example, splitting a source document into chunks for a RAG pipeline — and those children must be tracked independently.

One approach would be a central coordinator that determines completion by polling all components — querying the Connector, all Worker instances, and all Indexer instances, and declaring the run done when all queues are empty and all components report no work in flight. Lucille avoids this design: it would require every component to expose a status interface, and the coordinator would need to know the full topology of the running system. Instead, Lucille uses an event-driven model in which components report their own progress without being interrogated.

The Solution: an Event Queue and a Publisher

Lucille’s solution is to introduce two additional elements:

An event queue carries document lifecycle notifications from Workers and Indexers. When a document is successfully indexed, the Indexer sends a FINISH event. When a document fails, a FAIL event is sent. When a Stage deliberately drops a document, a DROP event is sent. When a Worker generates a child document, a CREATE event is sent so that child can be tracked.

A Publisher is used by the Connector to introduce documents into the system. The Publisher stamps each document with a run ID, records its ID as pending, and places it on the processing queue. It then listens to the event queue, reconciling events against its pending set. When a FINISH, FAIL, or DROP event arrives for a document ID, the Publisher removes it from the pending set. When a CREATE event arrives for a child document, the Publisher adds that new ID to the pending set.

The run is complete when three conditions hold simultaneously: the Connector has stopped, the event queue is empty, and the Publisher has no pending IDs. Any one condition alone is insufficient.

This design handles two subtle edge cases. First, a child document can be indexed and receive a FINISH event before the Publisher has processed its CREATE event, because Workers and Indexers run concurrently. The Publisher handles this with a secondary ledger of premature completions that it reconciles when the CREATE event eventually arrives. Second, the pending set is a Bag rather than a Set — two documents with the same ID can be published in the same run, and a Bag counts duplicates, so two publishes of the same ID require two separate terminal events to clear.

The complete pseudocode, now including the event queue and Publisher, is shown below. See Appendix: Full Pseudocode for the detailed version presented as an appendix.

Appendix: Full Pseudocode

// === QUEUES ===

Queue processingQueue;  // documents waiting to be enriched by Workers
Queue indexingQueue;    // processed documents waiting to be sent to the search backend
Queue eventQueue;       // document lifecycle events (CREATE, FINISH, FAIL, DROP)


// === RUNNER / CONNECTOR ===
// Runner launches the connector in its own thread; the main thread waits for completion.

publisher = new Publisher(processingQueue, eventQueue);

new Thread(() -> {
    // publisher.publish() is thread-safe, so a connector can spawn multiple
    // publishing threads; this example calls publish() sequentially
    while (source.hasNext()) {
        Document doc = Document.from(source.next());
        publisher.publish(doc);  // stamps run ID, tracks doc ID, puts on processingQueue
    }
}).start();

// main thread: block until all documents (and their children) reach a terminal state
publisher.waitForCompletion();


// === WORKER (N instances) ===
// Each instance has its own Pipeline with its own instances of every Stage.
// Stages can hold stateful resources (models, connections, compiled patterns)
// without synchronization because they are never shared across threads.

pipeline = new Pipeline(stages);  // per-thread instance

while (running) {
    Document doc = processingQueue.poll();    // blocking poll with timeout
    Iterator<Document> results = pipeline.process(doc);

    for (Document result : results) {
        if (result.isChild()) {
            eventQueue.send(CREATE, result);  // notify publisher of new child
        }
        if (result.isDropped()) {
            eventQueue.send(DROP, result);    // notify publisher, discard document
        } else {
            indexingQueue.put(result);        // send for indexing
        }
    }
}


// === INDEXER (M instances) ===
// Each instance runs independently, consuming from the shared indexingQueue.

while (running) {
    Document doc = indexingQueue.poll();      // blocking poll with timeout

    batch.add(doc);

    if (batch.isFull() || batch.isExpired()) {
        try {
            searchClient.bulkIndex(batch.flush());
            for (Document indexed : batch) {
                eventQueue.send(FINISH, indexed);
            }
        } catch (Exception e) {
            for (Document failed : batch) {
                eventQueue.send(FAIL, failed);
            }
        }
    }
}


// === PUBLISHER (event loop, runs on main thread) ===
// Reconciles lifecycle events to determine when all work is complete.

Bag<String> pending = {};  // IDs not yet in a terminal state

on publish(doc):
    pending.add(doc.id);

on event(CREATE, id):
    pending.add(id);

on event(FINISH | FAIL | DROP, id):
    pending.remove(id);

isComplete():
    return connector.isDone() && eventQueue.isEmpty() && pending.isEmpty();

Nothing in this pseudocode specifies what the queues actually are — whether they are in-memory data structures or distributed messaging infrastructure. That question is addressed in the next page: Pluggable Queueing and the Deployment Model.

2.1.3 - From Single Process to Distributed

How the same pipeline code runs identically in a single JVM or distributed across machines — and how the Messenger abstraction makes this possible.

The previous page showed how to parallelize search ingestion using three concurrent components — Connector, Workers, and Indexers — communicating through a processing queue and an indexing queue. What we have not yet discussed is what those queues actually are.

What a Queue Must Provide

The queues in Lucille’s architecture have a specific set of requirements. They must support concurrent access from multiple producers and consumers. They must support a blocking poll() operation so that a Worker or Indexer that finds an empty queue waits efficiently rather than spinning. And when multiple Workers are calling poll() concurrently, work must be distributed fairly among them — each document should go to exactly one Worker.

The Case Against Hardcoding

In a project like Lucille, it would have been tempting to choose a suitable queue implementation and hardcode it directly into the components that use it. But either obvious choice would have carried a significant cost.

Hardcoding LinkedBlockingQueue would limit scaling to what can be achieved by adding threads inside a single JVM. For large-scale ingestion that needs to be distributed across multiple machines, this ceiling is too low.

Hardcoding Kafka would require every user to have a running Kafka cluster — even for a simple ingest that does not need to scale beyond one JVM. For development, testing, and smaller production workloads, this is unnecessary infrastructure with real operational overhead.

The Key Observation: Queues Are Used in a Limited, Standardized Way

Lucille’s components interact with queues in a narrow and well-defined way: producers call put(), consumers call poll(). There is no complex queue-specific logic embedded in the component code — no partition management, no consumer group coordination, no topic configuration. Since the interface is this simple, the queue implementation can be made pluggable without changing any component code.

Lucille achieves this through a Messenger abstraction. Each component type has its own Messenger interface — PublisherMessenger, WorkerMessenger, IndexerMessenger — that defines exactly the messaging operations that component needs. The Connector and Publisher use a PublisherMessenger to put documents on the processing queue and read from the event queue. Workers use a WorkerMessenger to poll the processing queue and put results on the indexing queue. Indexers use an IndexerMessenger to poll the indexing queue and send events.

Each component is written entirely against its Messenger interface. The component code is identical regardless of which Messenger implementation is in use — it has no knowledge of whether the underlying queue is an in-memory data structure or a Kafka topic.

A Flexible Deployment Model and a Natural Scaling Path

The pluggable Messenger model directly enables a flexible deployment model.

For a simple deployment, the Connector, Workers, and Indexer all run as threads inside a single JVM where the queues are in-memory LinkedBlockingQueue instances with fixed capacity. This is the LocalMessenger implementation. No external infrastructure is required.

To scale the deployment, the Connector, Workers, and Indexers can run in separate JVMs where the queues are Kafka topics. This is the KafkaMessenger implementation. Kafka’s consumer group protocol handles partition assignment and rebalancing as Workers are added or removed.

This gives Lucille a natural scaling path. You can begin with a single-JVM deployment, gain experience with the system, tune your pipeline, and validate your configuration — all without Kafka. When your data volumes require more throughput than a single JVM can provide, you introduce Kafka and distribute the load across multiple machines. The only thing that changes in this transition is the Messenger implementation. Every other code path — the pipeline stages, the accounting logic, the retry behavior, the batching — remains exactly the same. This gives a high degree of confidence that the system will behave identically at scale to how it behaved in a single-JVM deployment.

Testability

The pluggable Messenger model also has significant advantages for testing.

Moving from a sequential loop to a concurrent, queue-based architecture introduces a new challenge: the system becomes harder to test. A simple loop with mocked externals runs trivially inside a unit test. A multi-threaded, queue-based system with concurrent components does not — unless it has been specifically designed to make testing straightforward.

The Messenger abstraction solves this directly. Lucille provides a test mode where a TestMessenger wraps the LocalMessenger and records a history of every message sent between components — every document published, every document sent for indexing, every lifecycle event. After a test run, test code can retrieve this history and make assertions about exactly what happened.

The Indexer runs in bypass mode during tests: the full indexing code path executes — batching, event sending, error handling — right up to the point of communicating with the search backend, which is bypassed. This means the Indexer’s logic is exercised without requiring a live Solr or OpenSearch instance.

Connectors and pipeline stages that communicate with live source systems do need some form of mocking for those external systems — for example, using WireMock to simulate an HTTP endpoint, or an in-memory database for a JDBC connector. But that mocking is scoped to the external system itself. The Lucille components — the Workers, the Indexer, the Publisher, the event queue — are not mocked at all. They run exactly as they would in production.

The practical effect is that end-to-end integration tests of complex pipelines are straightforward to write. The full Lucille system runs inside a unit test, in its complete and realistic form, without requiring Lucille’s own components to be stubbed out. The confidence this provides is significant: a pipeline that passes its tests in this mode has been proved to work with the real Lucille mechanics, not a simplified approximation of them.

The pages so far have focused on the batch ingest model — a bounded run with a defined start and end. Lucille also supports a streaming model for continuous, unbounded ingestion, as well as a hybrid WorkerIndexer deployment pattern that offers a practical middle ground between single-JVM and fully distributed. These are covered in the next page: Topology.

2.1.4 - Topology

Lucille can be configured to best support your use case.

Batch and Streaming Models

The pages so far have described Lucille in terms of a batch ingest: a bounded run where a Connector reads a finite dataset to completion, the Publisher tracks every document through to a terminal state, and the run ends when all work is done.

Lucille also supports a streaming model for unbounded, continuous ingestion. In a streaming scenario, documents arrive continuously from an external source and need to be processed and indexed as they arrive. There is no run boundary, no completion accounting, and no Connector lifecycle — the stream simply flows.

From an architectural standpoint, the streaming model is a straightforward variation of the batch model. Because Lucille is a queue-based system, the Connector’s role can be taken over by an external producer that writes documents directly to the processing queue (a Kafka topic). Workers consume from that topic and process documents exactly as they would in batch mode. The rest of the system — Workers writing to an indexing queue, Indexers reading from it and sending batches to the search backend — functions identically.

Crucially, the pipeline stages themselves are unaware of which model they are running under. A Stage that extracts entities or generates embeddings does not know or care whether it is processing a bounded batch or an unbounded stream. This means you can develop and test pipeline logic in batch mode — where the accounting and test infrastructure make correctness verification straightforward — and then deploy the same pipeline in streaming mode for real-time production ingestion.

The WorkerIndexer: A Practical Middle Ground

In the fully distributed deployment, Workers and Indexers run as separate JVM processes connected by a Kafka topic. This maximizes flexibility — each fleet can be scaled independently — but it adds a Kafka round-trip between Worker output and Indexer input.

As a practical simplification, Lucille provides the WorkerIndexer: a single JVM process that pairs a Worker with a co-located Indexer. The Worker reads from Kafka and writes processed documents to an in-memory queue; the paired Indexer reads from that in-memory queue and sends to the search backend. This eliminates the Worker-to-Indexer Kafka round-trip while retaining horizontal scaling — you can run as many WorkerIndexer processes as needed, and Kafka’s consumer group protocol distributes partitions across them automatically.

The WorkerIndexer is a useful middle ground between the fully local single-JVM deployment and the fully distributed model with independent Worker and Indexer fleets. It is the recommended starting point for distributed deployments that do not require separate Worker and Indexer scaling.

Worker Processes and Worker Threads

There are two independent knobs for scaling enrichment throughput, and it is important to distinguish them.

A Worker process is a JVM process running Lucille’s Worker component. In distributed deployments, multiple Worker processes can be launched on separate machines, all consuming from the shared Kafka processing topic.

Within each Worker process, a Worker thread is a single thread executing the pipeline — and each Worker thread is also an independent Kafka consumer. The Kafka poll() call happens inside the thread, so each thread fetches and processes documents directly from the topic. Kafka’s consumer group protocol assigns partitions across all active consumer threads, summed across all Worker processes. A Worker process runs a configurable number of Worker threads (worker.threads), each with its own independent Pipeline instance — its own copy of every Stage, with its own connections, models, and any other stateful resources. Because Stage instances are never shared across threads, they require no synchronization.

The two knobs compose: to maximize throughput, scale out with more Worker processes across machines, and scale up with more Worker threads within each process. One constraint applies: the total number of Worker threads across all processes should not exceed the number of partitions in the Kafka source topic, since Kafka cannot assign a partition to more than one consumer in the same group — excess threads would sit idle. The same distinction applies to WorkerIndexer processes, where each process contains a Worker thread pool paired with an Indexer.

For the commands to start each mode in production, see Production Deployment.

Local Modes

In local modes (also called single-node or standalone modes), all Lucille components run as threads in a single JVM process. Lucille supports two local modes.

Local

All components run as threads in one JVM. Inter-component communication uses in-memory queues. No external dependencies required.

Architecture diagram: local mode

Kafka Local

All components still run as threads in one JVM, but inter-component communication uses an external Kafka instance instead of in-memory queues. Useful for testing Kafka integration without deploying separate processes.

Architecture diagram: Kafka-local mode

Distributed Modes

Distributed modes are how Lucille scales horizontally. Kafka provides message persistence and fault tolerance, and each Lucille component type runs as one or more separate JVM processes.

Fully Distributed

Best for batch ingest architecture.

The Connector/Publisher, Workers, and Indexers each run as separate JVM processes. All inter-component communication flows through Kafka topics. Workers and Indexers send lifecycle events to a Kafka event topic, which the Publisher polls to track completion.

Architecture diagram: fully distributed mode

Connector-less Distributed

Best for streaming ingest architecture.

Like Fully Distributed, but with no Lucille Connector or Publisher. An external system writes documents directly to the Kafka source topic, where Workers pick them up. Because there is no Publisher, there is no run-completion accounting — this mode is intended for unbounded streaming workloads.

If events are enabled, the external publisher must stamp each document with a run_id.

Architecture diagram: connector-less distributed mode

Hybrid

Best for streaming update architecture.

Like Connector-less Distributed, but each Worker is paired with a co-located Indexer in the same JVM (a WorkerIndexer). Workers read from a Kafka source topic and write processed documents to an in-memory queue; the paired Indexer reads from that queue and sends to the search backend. This eliminates one Kafka round-trip per document compared to Fully Distributed.

Architecture diagram: hybrid mode

2.1.5 - Document Lifecycle

The complete journey of a single Document through Lucille, from raw data in a source system to a searchable record in the search backend.

This page traces a single Document from the moment it exists only as raw data in a source system to the moment it is retrievable by a search query. It takes the Lucille architecture as a given — for the conceptual explanation of how the system is structured, see Parallelizing Search ETL and Pluggable Queueing.

The Happy Path

Before a document’s lifecycle begins, the system components must be set up. A batch ingest is launched via the Runner, which first validates the configuration. Assuming a single-connector config, the Runner creates a Publisher, passes it to the Connector, and launches the Connector in its own thread. The Runner then calls publisher.waitForCompletion() and blocks. In distributed mode, the Worker and Indexer processes would have been started separately beforehand. In local mode, the Runner starts the Worker and Indexer as threads in the same JVM. With this infrastructure in place, documents can begin their journey.

Step 1 — Raw data in the source system

The document begins as a row in a database table, a file on a filesystem, a message on a Kafka topic, or any other source that Lucille’s Connector knows how to read. At this point it is not yet a Lucille Document — it is raw data in whatever format the source system uses.

Step 2 — The Connector reads the record and creates a Document

Inside connector.execute(publisher), the Connector reads the next record from its source and constructs a Document with a stable, user-defined ID. The ID must be deterministic — if the pipeline is re-run, the same source record must produce the same document ID so that the Indexer’s upsert semantics correctly update rather than duplicate the record in the search backend.

The Connector calls publisher.publish(document).

Step 3 — The Publisher registers the Document and puts it on the processing queue

Before placing the document on the processing queue, the Publisher does two things in order:

Stamps the document with the run_id — an immutable field set once and used for log correlation, event routing, and Kafka topic naming throughout the document’s lifetime.
Registers the document’s ID in its accounting ledger (docIdsToTrack). Registration happens before the document is placed on the queue — the moment it is on the queue, a Worker could process it and the Indexer could send a FINISH event. If the Publisher hadn’t registered the ID yet, it would misclassify that event.

The document is then placed on the processing queue. In distributed mode, this crosses a Kafka boundary: the document becomes a message on the {pipeline}_source Kafka topic, serialized and persisted.

No lifecycle event is sent at this point — the Publisher tracks the document internally through the accounting ledger rather than via the event queue.

Step 4 — A Worker thread picks up the Document

A Worker thread is blocking on processingQueue.poll(). In distributed mode, this is a Kafka consumer poll() call — each Worker thread is an independent Kafka consumer within the consumer group, and Kafka’s consumer group protocol has assigned it one or more partitions of the source topic. The document is delivered to exactly one Worker thread.

The Worker passes the document through each Stage in the pipeline in sequence.

Step 5 — Pipeline processing

Each Stage receives the document, performs its transformation — extracting text, running NLP, generating an embedding, querying a database — and returns the result.

Step 6 — The parent Document moves to the indexing queue

Once the parent document has passed through all pipeline stages, the Worker places it on the indexing queue. In distributed mode, this crosses a second Kafka boundary: the document becomes a message on the {pipeline}_dest Kafka topic.

Step 7 — The Indexer batches and sends the Document

The Indexer thread is blocking on indexingQueue.poll(). It accumulates documents in a batch. When the batch reaches its configured size (indexer.batchSize, default 100 documents) or its timeout elapses (indexer.batchTimeout, default 100ms), the Indexer issues a single bulk API call to the search backend.

Before sending, the Indexer strips reserved internal fields (___dropped, ___skipped, ___children) and applies any configured field whitelist or blacklist. The document that reaches the search backend contains only the fields you want indexed.

The search backend accepts the document. The document now exists in the index.

Step 8 — The Indexer sends a FINISH event

After a successful bulk operation, the Indexer sends a FINISH event to the event queue for each document in the batch — including our document. In distributed mode this event travels on the {pipeline}_event_{runId} Kafka topic, which is unique per run so that events from concurrent runs never interfere with each other.

The Publisher’s polling loop receives the FINISH event, looks up the document’s ID in docIdsToTrack, and removes it. The document’s lifecycle is complete.

Step 9 — The Document is searchable

The document has been accepted by the search backend. Once the backend makes it visible — via a scheduled commit in Solr, or after the refresh interval in Elasticsearch and OpenSearch — a query can retrieve it. Lucille does not issue commits; visibility timing is controlled entirely by the search backend’s configuration. The enrichment performed during pipeline processing — the extracted text, the entity tags, the vector embedding — is available for full-text search, faceting, and semantic retrieval.

After all documents (parents and children) have completed their individual lifecycles, the framework determines that the run is complete. The Publisher continuously evaluates three conditions: (1) the Connector thread has terminated (no more documents will be published), (2) the event queue is empty (no more events are in transit), and (3) docIdsToTrack is empty (every document has reached a terminal state). All three must hold simultaneously. When they do, waitForCompletion() returns and the Runner logs the run summary.

The Detailed Path

The Happy Path above omits edge cases, error handling, logging, metrics, backpressure, routing, and child documents. This section walks through the same journey again, annotating each step with everything that actually happens. A developer debugging “why didn’t my document get indexed?” can walk through these steps to identify where the document’s journey diverged.