This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Overview

A narrative introduction to Lucille’s architecture — the problem it solves, the core design, and how documents flow through the system.

1: The Problem of Search ETL
2: Parallelizing Search ETL
3: From Single Process to Distributed
4: Topology

5: Document Lifecycle
6: Design Rationale

Read these pages in order for a complete introduction to Lucille’s architectural design.

The Problem of Search ETL — Why a simple sequential ingest loop falls short in production and what needs to change.
Parallelizing Search ETL — How Lucille maps the three ETL functions to concurrent components communicating through queues, and how it tracks document lifecycle across an asynchronous system.
From Single Process to Distributed — How the same pipeline code runs identically in a single JVM or distributed across machines, and how the Messenger abstraction makes this possible.
Topology — Batch and streaming models, the WorkerIndexer, and deployment configurations from single-JVM to fully distributed.
Document Lifecycle — The complete journey of a single Document through the system, plus cross-cutting concerns like logging, metrics, and testing.
Design Rationale — The 24 requirements that govern Lucille’s architecture and why the system is designed the way it is.

1 - The Problem of Search ETL

Why a simple sequential loop falls short for production search ingestion, and how Lucille addresses the pitfalls.

Organizations build search applications when their users need to find information across large, heterogeneous collections of data — product catalogs, legal documents, support tickets, research papers, internal wikis, file shares. The data lives in databases, cloud storage, APIs, and content management systems, often in different formats and with varying levels of structure. Before any of it becomes searchable, it must be extracted from these sources, cleaned, enriched (with metadata, classifications, embeddings, or extracted entities), and delivered to a search engine or vector database in the right format. This process — getting source data into a search-ready state — is search ETL.

Getting data into a search engine involves three distinct functions:

Connect to a source system to acquire data.
Clean and enrich the data.
Send the data to the search backend.

In a first prototype of a search ingestion process, it is common to perform these functions in a sequential loop:

// as long as the source system has more data for us to consume…
while (source.hasNext()) {
    // 1 – create a Document from the next record
    Document doc = Document.from(source.next());
    // 2 – pass the Document through an enrichment pipeline
    pipeline.process(doc);
    // 3 – send the document to the search engine
    searchClient.index(doc);
}

When you’re ingesting a few hundred documents in a POC, this loop finishes in seconds. It works. It’s easy to reason about. At this point, adopting a search ETL framework feels like overkill.

Then you go to production.

The dataset is no longer a few hundred documents — it’s millions. The loop that finished in ten seconds now takes a full day. And over the course of that day, you run into a series of problems that the simple loop has no answer for.

1. One bad document stops everything

Somewhere in that day-long run, a document fails. Maybe the enrichment logic throws an exception on an unexpected input. Maybe the search backend rejects a malformed field. The loop stops.

Do you fix the document and restart from the beginning? That means re-ingesting everything you already indexed. Do you build checkpointing logic so you can resume from where you left off? Now you need a persistent state store. Or do you skip the bad document and keep going? If you do, where does the failed document go? How do you know at the end of the run which documents succeeded and which didn’t? How do you inspect and reprocess just the failures?

The simple loop gives you no framework for any of these decisions. And search backends make this harder than general ETL in a specific way: bulk indexing APIs return mixed results. A single bulk request can partially succeed — some documents accepted, others individually rejected due to a field type mismatch or a mapping conflict — and the response requires per-document inspection to know what actually happened. You can’t treat indexing as a simple pass/fail operation and retry the batch whole. You need per-document error tracking inside every batch.

2. You can’t see what’s happening

The run has been going for six hours. Is it on track? Is it stuck? Are there bottlenecks building up somewhere? How many documents have been indexed so far, and how many are still waiting?

The loop has no instrumentation. You have no visibility into throughput, no per-stage latency breakdown, no count of in-flight documents, no way to distinguish “slow but progressing” from “silently stuck.” You’re watching a process run for a day with no dashboard and no progress bar.

3. Enrichment logic becomes tangled and unreusable

As the pipeline grows — adding text extraction, NLP, embedding generation, database lookups — the enrichment logic accumulates inside or alongside the loop. It becomes difficult to adjust individual steps without touching the whole thing. Testing a change requires running the full ingest or building a separate harness.

When a second project needs some of the same transformations, there’s no clean mechanism for reuse. The logic gets copied or extracted into a shared library that gradually accumulates its own complexity. Configuration for which steps run and in what order is hardcoded rather than declared.

4. One source document fans out into many index records

In general ETL, a record in the source maps to a record in the destination. Search ingestion breaks that assumption.

For semantic search and RAG pipelines, a single source document — a PDF, a support ticket, a product page — often needs to be split into smaller chunks, each indexed as an independent search record so that retrieval can return the most relevant passage rather than the entire document. One source document becomes tens or hundreds of index records.

The simple loop has no model for this. searchClient.index(doc) indexes one document. If a stage in the pipeline splits a document into chunks, those chunks need to flow through the rest of the pipeline independently, be tracked individually for accounting purposes, and be indexed as separate records. The parent document and its chunks may also need to be indexed in a specific order if the chunks reference the parent by ID.

This 1-to-N fan-out is unique to search ingestion. It doesn’t arise in database ETL, and it breaks both the loop’s execution model and any completion-tracking logic built on top of it.

5. Environment-specific settings are hardcoded

The pipeline runs against a development search backend today. Next week it needs to run against staging, with different credentials, a different Solr URL, and a different collection name. After that, production — with its own connection details, possibly different Kafka brokers, and secrets that shouldn’t be hardcoded.

Managing that across environments — without duplicating code, without accidentally running dev config against prod, with a clear way to inject secrets at runtime — is not a problem the loop was designed to solve.

6. The sequential loop can’t go faster

This is the hardest problem, and the reason all the others matter more once you solve it.

If a full dataset ingest takes a day, the obvious question is: can we make it faster? The answer is yes — but not by iterating on the sequential loop. To go faster, you have to parallelize.

Part of what makes this problem harder in search ingestion than in general ETL is the cost of the enrichment step itself. In database-to-database ETL, transformation is often lightweight — type conversion, field mapping, simple lookups. In search ingestion, enrichment exists specifically to pre-compute things that make search fast and relevant: OCR on scanned documents, named entity extraction, classification, vector embedding generation. This work is genuinely expensive per document. The pipeline is CPU-bound in a way that most ETL pipelines are not, which is exactly why parallelizing it matters so much — and why the gains from doing it correctly are so large.

The loop is slow because its three steps have fundamentally different performance characteristics: connecting to the source system is I/O-bound, pipeline enrichment is often CPU-bound (model inference, API calls, computation), and indexing is I/O-bound but also depends on the CPU of the search backend. The sequential loop adds the latency of all three steps per document:

total time = ((source latency) + (pipeline latency) + (indexing latency)) × (number of documents)

To do better, you need to run these three phases concurrently — reading the next batch of documents while the previous batch is being enriched, while the batch before that is being indexed. And you need multiple enrichment workers running in parallel to saturate available CPU.

But parallelizing the loop creates its own roadblocks. Multiple reader threads need to coordinate so they don’t fetch the same records twice. The enrichment stage may hold expensive resources — models, connection pools — that can’t safely be shared across threads. The search backend performs best when documents arrive in batches, not one at a time, which requires accumulating results from multiple parallel workers before flushing. And if you scale to multiple machines, you need a message queue to distribute work across processes, which means managing Kafka topics, consumer groups, and partition assignment.

Then, once you succeed in parallelizing, a new set of problems emerges. How do you know when the run is complete? With a sequential loop, the answer is obvious: the loop returns. With parallel workers, you can’t know the run is done just because the source is exhausted — documents are still in flight, being processed and indexed. You need an accounting system that tracks every document from publication through its terminal state. How do you handle backpressure? A fast source feeding slow enrichment workers will produce documents faster than they can be consumed, eventually exhausting memory. How do you handle a worker crash mid-run? Kafka’s consumer group protocol can reassign its partitions, but unacknowledged documents need to be redelivered without silently dropping or double-indexing them. What about a document that repeatedly crashes a worker — a malformed input that triggers a bug in third-party code? Without a retry limit and a dead-letter queue, one bad document can loop forever.

Lucille is the system you arrive at after encountering these problems in real production deployments, solving them one by one, and validating the solutions at scale. It is not a theoretical framework — it is the accumulated result of decisions forced by real failure modes.

2 - Parallelizing Search ETL

How Lucille structures search ingestion as three concurrent components communicating through queues, and how it tracks document lifecycle across an asynchronous system.

Lucille addresses all of the pain points described in The Problem of Search ETL — error handling, observability, pipeline composition, fan-out, configuration management, and scalability. This page focuses on the last and most fundamental of those problems: the sequential loop can’t go faster.

Lucille’s answer is a parallel architecture whose components derive directly from the structure of the problem. Each of the three functions of search ETL — acquiring data, enriching it, and indexing it — becomes its own independent concurrent component, operating at its own pace, limited only by its own resources rather than by the speed of the others.

To design any concurrent system, two questions must be answered:

What are the core components?
How do they communicate?

The Core Components

Connectors connect to source systems to acquire data. A Connector reads from its source — a filesystem, a database, a Kafka topic, an RSS feed — and emits Documents into the system one at a time. It does not know how many Workers will process those documents, or how long enrichment will take.

Workers enrich documents by passing them through a Pipeline of Stages. Each Stage performs a specific transformation: extracting text, running NLP, generating embeddings, looking up database records. A Worker handles one document at a time and writes the result onward. Multiple Workers run concurrently, each with its own Pipeline instance, so enrichment scales with available CPU.

Indexers receive processed documents and send them in batches to the search backend. An Indexer accumulates documents until a batch is full or a timeout expires, then issues a single bulk API call. Batching is essential for search engine performance — bulk writes are significantly faster than one-at-a-time indexing.

No component waits for another unless it exhausts its backlog of work — or the downstream queue is full, which provides natural backpressure on the upstream component.

How Should They Communicate?

In some distributed systems, asynchronous components require tight, fine-grained communication. Consider a distributed query engine where a central coordinator assigns specific data partitions to worker nodes, monitors their progress with heartbeats, detects straggler tasks, re-assigns failed partitions to other nodes, and waits for explicit acknowledgment from every worker before declaring a query complete. The coordinator and workers exchange a continuous stream of control messages — assignment, acknowledgment, status updates, failure notifications — and the coordinator must maintain a detailed model of the state of every worker at all times.

Lucille takes a different approach. The key observation is that Connectors, Workers, and Indexers can be largely decoupled. No component needs to know what the others are doing in real time. A Connector doesn’t need to know how many Workers exist or whether they are fast or slow — it just needs somewhere to put its output. A Worker doesn’t need to know which Connector produced a document or which Indexer will consume it. An Indexer doesn’t need to know where documents came from or how many are still in the pipeline behind them.

This means it suffices for the components to communicate via queues. Each component places its output on a queue for the next component to read from:

The Connector puts documents on a processing queue.
Workers read the processing queue, process documents through the Pipeline, and put results on an indexing queue.
Indexers read the indexing queue and send documents to the search backend.

No component calls another directly. No component waits for a response. Each simply produces to a queue and consumes from a queue. The queue is the entire communication contract.

// Shared queues connecting the three components.
Queue processingQueue;   // documents waiting to be enriched
Queue indexingQueue;     // enriched documents waiting to be indexed


// === CONNECTOR ===
while (source.hasNext()) {
    Document doc = Document.from(source.next());
    processingQueue.put(doc);
}


// === WORKER (N instances, running concurrently) ===
while (running) {
    Document doc = processingQueue.poll();
    Document result = pipeline.process(doc);
    indexingQueue.put(result);
}


// === INDEXER (M instances, running concurrently) ===
while (running) {
    Document doc = indexingQueue.poll();
    batch.add(doc);
    if (batch.isFull() || batch.isExpired()) {
        searchClient.bulkIndex(batch.flush());
    }
}

This is the core of Lucille’s architecture. The Connector, Workers, and Indexers need no knowledge of each other beyond the existence of these two queues.

What’s Missing?

The pseudocode above captures the essential structure but leaves out something important: there is no way to know when a batch ingest is complete, no way to track how many documents have succeeded or failed, and no support for two common pipeline operations — dropping documents that should not be indexed, and emitting child documents from a single source record.

With the sequential loop, completion was trivial:

while (source.hasNext()) { }
// loop returns → run is done

In a concurrent system, the Connector finishing is not enough. When the Connector stops putting documents on the processing queue, Workers may still be enriching documents, and Indexers may still be sending batches. The system is not done until every document has reached a terminal state — indexed, failed, or dropped.

Tracking this requires knowing about every document in flight: which ones have been introduced, which have completed, and which are still being processed. A Worker can also generate new child documents during pipeline processing — for example, splitting a source document into chunks for a RAG pipeline — and those children must be tracked independently.

One approach would be a central coordinator that determines completion by polling all components — querying the Connector, all Worker instances, and all Indexer instances, and declaring the run done when all queues are empty and all components report no work in flight. Lucille avoids this design: it would require every component to expose a status interface, and the coordinator would need to know the full topology of the running system. Instead, Lucille uses an event-driven model in which components report their own progress without being interrogated.

The Solution: an Event Queue and a Publisher

Lucille’s solution is to introduce two additional elements:

An event queue carries document lifecycle notifications from Workers and Indexers. When a document is successfully indexed, the Indexer sends a FINISH event. When a document fails, a FAIL event is sent. When a Stage deliberately drops a document, a DROP event is sent. When a Worker generates a child document, a CREATE event is sent so that child can be tracked.

A Publisher is used by the Connector to introduce documents into the system. The Publisher stamps each document with a run ID, records its ID as pending, and places it on the processing queue. It then listens to the event queue, reconciling events against its pending set. When a FINISH, FAIL, or DROP event arrives for a document ID, the Publisher removes it from the pending set. When a CREATE event arrives for a child document, the Publisher adds that new ID to the pending set.

The run is complete when three conditions hold simultaneously: the Connector has stopped, the event queue is empty, and the Publisher has no pending IDs. Any one condition alone is insufficient.

This design handles two subtle edge cases. First, a child document can be indexed and receive a FINISH event before the Publisher has processed its CREATE event, because Workers and Indexers run concurrently. The Publisher handles this with a secondary ledger of premature completions that it reconciles when the CREATE event eventually arrives. Second, the pending set is a Bag rather than a Set — two documents with the same ID can be published in the same run, and a Bag counts duplicates, so two publishes of the same ID require two separate terminal events to clear.

The complete pseudocode, now including the event queue and Publisher, is shown below. See Appendix: Full Pseudocode for the detailed version presented as an appendix.

Appendix: Full Pseudocode

// === QUEUES ===

Queue processingQueue;  // documents waiting to be enriched by Workers
Queue indexingQueue;    // processed documents waiting to be sent to the search backend
Queue eventQueue;       // document lifecycle events (CREATE, FINISH, FAIL, DROP)


// === RUNNER / CONNECTOR ===
// Runner launches the connector in its own thread; the main thread waits for completion.

publisher = new Publisher(processingQueue, eventQueue);

new Thread(() -> {
    // publisher.publish() is thread-safe, so a connector can spawn multiple
    // publishing threads; this example calls publish() sequentially
    while (source.hasNext()) {
        Document doc = Document.from(source.next());
        publisher.publish(doc);  // stamps run ID, tracks doc ID, puts on processingQueue
    }
}).start();

// main thread: block until all documents (and their children) reach a terminal state
publisher.waitForCompletion();


// === WORKER (N instances) ===
// Each instance has its own Pipeline with its own instances of every Stage.
// Stages can hold stateful resources (models, connections, compiled patterns)
// without synchronization because they are never shared across threads.

pipeline = new Pipeline(stages);  // per-thread instance

while (running) {
    Document doc = processingQueue.poll();    // blocking poll with timeout
    Iterator<Document> results = pipeline.process(doc);

    for (Document result : results) {
        if (result.isChild()) {
            eventQueue.send(CREATE, result);  // notify publisher of new child
        }
        if (result.isDropped()) {
            eventQueue.send(DROP, result);    // notify publisher, discard document
        } else {
            indexingQueue.put(result);        // send for indexing
        }
    }
}


// === INDEXER (M instances) ===
// Each instance runs independently, consuming from the shared indexingQueue.

while (running) {
    Document doc = indexingQueue.poll();      // blocking poll with timeout

    batch.add(doc);

    if (batch.isFull() || batch.isExpired()) {
        try {
            searchClient.bulkIndex(batch.flush());
            for (Document indexed : batch) {
                eventQueue.send(FINISH, indexed);
            }
        } catch (Exception e) {
            for (Document failed : batch) {
                eventQueue.send(FAIL, failed);
            }
        }
    }
}


// === PUBLISHER (event loop, runs on main thread) ===
// Reconciles lifecycle events to determine when all work is complete.

Bag<String> pending = {};  // IDs not yet in a terminal state

on publish(doc):
    pending.add(doc.id);

on event(CREATE, id):
    pending.add(id);

on event(FINISH | FAIL | DROP, id):
    pending.remove(id);

isComplete():
    return connector.isDone() && eventQueue.isEmpty() && pending.isEmpty();

Nothing in this pseudocode specifies what the queues actually are — whether they are in-memory data structures or distributed messaging infrastructure. That question is addressed in the next page: Pluggable Queueing and the Deployment Model.

3 - From Single Process to Distributed

How the same pipeline code runs identically in a single JVM or distributed across machines — and how the Messenger abstraction makes this possible.

The previous page showed how to parallelize search ingestion using three concurrent components — Connector, Workers, and Indexers — communicating through a processing queue and an indexing queue. What we have not yet discussed is what those queues actually are.

What a Queue Must Provide

The queues in Lucille’s architecture have a specific set of requirements. They must support concurrent access from multiple producers and consumers. They must support a blocking poll() operation so that a Worker or Indexer that finds an empty queue waits efficiently rather than spinning. And when multiple Workers are calling poll() concurrently, work must be distributed fairly among them — each document should go to exactly one Worker.

The Case Against Hardcoding

In a project like Lucille, it would have been tempting to choose a suitable queue implementation and hardcode it directly into the components that use it. But either obvious choice would have carried a significant cost.

Hardcoding LinkedBlockingQueue would limit scaling to what can be achieved by adding threads inside a single JVM. For large-scale ingestion that needs to be distributed across multiple machines, this ceiling is too low.

Hardcoding Kafka would require every user to have a running Kafka cluster — even for a simple ingest that does not need to scale beyond one JVM. For development, testing, and smaller production workloads, this is unnecessary infrastructure with real operational overhead.

The Key Observation: Queues Are Used in a Limited, Standardized Way

Lucille’s components interact with queues in a narrow and well-defined way: producers call put(), consumers call poll(). There is no complex queue-specific logic embedded in the component code — no partition management, no consumer group coordination, no topic configuration. Since the interface is this simple, the queue implementation can be made pluggable without changing any component code.

Lucille achieves this through a Messenger abstraction. Each component type has its own Messenger interface — PublisherMessenger, WorkerMessenger, IndexerMessenger — that defines exactly the messaging operations that component needs. The Connector and Publisher use a PublisherMessenger to put documents on the processing queue and read from the event queue. Workers use a WorkerMessenger to poll the processing queue and put results on the indexing queue. Indexers use an IndexerMessenger to poll the indexing queue and send events.

Each component is written entirely against its Messenger interface. The component code is identical regardless of which Messenger implementation is in use — it has no knowledge of whether the underlying queue is an in-memory data structure or a Kafka topic.

A Flexible Deployment Model and a Natural Scaling Path

The pluggable Messenger model directly enables a flexible deployment model.

For a simple deployment, the Connector, Workers, and Indexer all run as threads inside a single JVM where the queues are in-memory LinkedBlockingQueue instances with fixed capacity. This is the LocalMessenger implementation. No external infrastructure is required.

To scale the deployment, the Connector, Workers, and Indexers can run in separate JVMs where the queues are Kafka topics. This is the KafkaMessenger implementation. Kafka’s consumer group protocol handles partition assignment and rebalancing as Workers are added or removed.

This gives Lucille a natural scaling path. You can begin with a single-JVM deployment, gain experience with the system, tune your pipeline, and validate your configuration — all without Kafka. When your data volumes require more throughput than a single JVM can provide, you introduce Kafka and distribute the load across multiple machines. The only thing that changes in this transition is the Messenger implementation. Every other code path — the pipeline stages, the accounting logic, the retry behavior, the batching — remains exactly the same. This gives a high degree of confidence that the system will behave identically at scale to how it behaved in a single-JVM deployment.

Testability

The pluggable Messenger model also has significant advantages for testing.

Moving from a sequential loop to a concurrent, queue-based architecture introduces a new challenge: the system becomes harder to test. A simple loop with mocked externals runs trivially inside a unit test. A multi-threaded, queue-based system with concurrent components does not — unless it has been specifically designed to make testing straightforward.

The Messenger abstraction solves this directly. Lucille provides a test mode where a TestMessenger wraps the LocalMessenger and records a history of every message sent between components — every document published, every document sent for indexing, every lifecycle event. After a test run, test code can retrieve this history and make assertions about exactly what happened.

The Indexer runs in bypass mode during tests: the full indexing code path executes — batching, event sending, error handling — right up to the point of communicating with the search backend, which is bypassed. This means the Indexer’s logic is exercised without requiring a live Solr or OpenSearch instance.

Connectors and pipeline stages that communicate with live source systems do need some form of mocking for those external systems — for example, using WireMock to simulate an HTTP endpoint, or an in-memory database for a JDBC connector. But that mocking is scoped to the external system itself. The Lucille components — the Workers, the Indexer, the Publisher, the event queue — are not mocked at all. They run exactly as they would in production.

The practical effect is that end-to-end integration tests of complex pipelines are straightforward to write. The full Lucille system runs inside a unit test, in its complete and realistic form, without requiring Lucille’s own components to be stubbed out. The confidence this provides is significant: a pipeline that passes its tests in this mode has been proved to work with the real Lucille mechanics, not a simplified approximation of them.

The pages so far have focused on the batch ingest model — a bounded run with a defined start and end. Lucille also supports a streaming model for continuous, unbounded ingestion, as well as a hybrid WorkerIndexer deployment pattern that offers a practical middle ground between single-JVM and fully distributed. These are covered in the next page: Topology.

4 - Topology

Lucille can be configured to best support your use case.

Batch and Streaming Models

The pages so far have described Lucille in terms of a batch ingest: a bounded run where a Connector reads a finite dataset to completion, the Publisher tracks every document through to a terminal state, and the run ends when all work is done.

Lucille also supports a streaming model for unbounded, continuous ingestion. In a streaming scenario, documents arrive continuously from an external source and need to be processed and indexed as they arrive. There is no run boundary, no completion accounting, and no Connector lifecycle — the stream simply flows.

From an architectural standpoint, the streaming model is a straightforward variation of the batch model. Because Lucille is a queue-based system, the Connector’s role can be taken over by an external producer that writes documents directly to the processing queue (a Kafka topic). Workers consume from that topic and process documents exactly as they would in batch mode. The rest of the system — Workers writing to an indexing queue, Indexers reading from it and sending batches to the search backend — functions identically.

Crucially, the pipeline stages themselves are unaware of which model they are running under. A Stage that extracts entities or generates embeddings does not know or care whether it is processing a bounded batch or an unbounded stream. This means you can develop and test pipeline logic in batch mode — where the accounting and test infrastructure make correctness verification straightforward — and then deploy the same pipeline in streaming mode for real-time production ingestion.

The WorkerIndexer: A Practical Middle Ground

In the fully distributed deployment, Workers and Indexers run as separate JVM processes connected by a Kafka topic. This maximizes flexibility — each fleet can be scaled independently — but it adds a Kafka round-trip between Worker output and Indexer input.

As a practical simplification, Lucille provides the WorkerIndexer: a single JVM process that pairs a Worker with a co-located Indexer. The Worker reads from Kafka and writes processed documents to an in-memory queue; the paired Indexer reads from that in-memory queue and sends to the search backend. This eliminates the Worker-to-Indexer Kafka round-trip while retaining horizontal scaling — you can run as many WorkerIndexer processes as needed, and Kafka’s consumer group protocol distributes partitions across them automatically.

The WorkerIndexer is a useful middle ground between the fully local single-JVM deployment and the fully distributed model with independent Worker and Indexer fleets. It is the recommended starting point for distributed deployments that do not require separate Worker and Indexer scaling.

Worker Processes and Worker Threads

There are two independent knobs for scaling enrichment throughput, and it is important to distinguish them.

A Worker process is a JVM process running Lucille’s Worker component. In distributed deployments, multiple Worker processes can be launched on separate machines, all consuming from the shared Kafka processing topic.

Within each Worker process, a Worker thread is a single thread executing the pipeline — and each Worker thread is also an independent Kafka consumer. The Kafka poll() call happens inside the thread, so each thread fetches and processes documents directly from the topic. Kafka’s consumer group protocol assigns partitions across all active consumer threads, summed across all Worker processes. A Worker process runs a configurable number of Worker threads (worker.threads), each with its own independent Pipeline instance — its own copy of every Stage, with its own connections, models, and any other stateful resources. Because Stage instances are never shared across threads, they require no synchronization.

The two knobs compose: to maximize throughput, scale out with more Worker processes across machines, and scale up with more Worker threads within each process. One constraint applies: the total number of Worker threads across all processes should not exceed the number of partitions in the Kafka source topic, since Kafka cannot assign a partition to more than one consumer in the same group — excess threads would sit idle. The same distinction applies to WorkerIndexer processes, where each process contains a Worker thread pool paired with an Indexer.

For the commands to start each mode in production, see Production Deployment.

Local Modes

In local modes (also called single-node or standalone modes), all Lucille components run as threads in a single JVM process. Lucille supports two local modes.

Local

All components run as threads in one JVM. Inter-component communication uses in-memory queues. No external dependencies required.

Architecture diagram: local mode

Kafka Local

All components still run as threads in one JVM, but inter-component communication uses an external Kafka instance instead of in-memory queues. Useful for testing Kafka integration without deploying separate processes.

Architecture diagram: Kafka-local mode

Distributed Modes

Distributed modes are how Lucille scales horizontally. Kafka provides message persistence and fault tolerance, and each Lucille component type runs as one or more separate JVM processes.

Fully Distributed

Best for batch ingest architecture.

The Connector/Publisher, Workers, and Indexers each run as separate JVM processes. All inter-component communication flows through Kafka topics. Workers and Indexers send lifecycle events to a Kafka event topic, which the Publisher polls to track completion.

Architecture diagram: fully distributed mode

Connector-less Distributed

Best for streaming ingest architecture.

Like Fully Distributed, but with no Lucille Connector or Publisher. An external system writes documents directly to the Kafka source topic, where Workers pick them up. Because there is no Publisher, there is no run-completion accounting — this mode is intended for unbounded streaming workloads.

If events are enabled, the external publisher must stamp each document with a run_id.

Architecture diagram: connector-less distributed mode

Hybrid

Best for streaming update architecture.

Like Connector-less Distributed, but each Worker is paired with a co-located Indexer in the same JVM (a WorkerIndexer). Workers read from a Kafka source topic and write processed documents to an in-memory queue; the paired Indexer reads from that queue and sends to the search backend. This eliminates one Kafka round-trip per document compared to Fully Distributed.

Architecture diagram: hybrid mode

5 - Document Lifecycle

The complete journey of a single Document through Lucille, from raw data in a source system to a searchable record in the search backend.

This page traces a single Document from the moment it exists only as raw data in a source system to the moment it is retrievable by a search query. It takes the Lucille architecture as a given — for the conceptual explanation of how the system is structured, see Parallelizing Search ETL and Pluggable Queueing.

The Happy Path

Before a document’s lifecycle begins, the system components must be set up. A batch ingest is launched via the Runner, which first validates the configuration. Assuming a single-connector config, the Runner creates a Publisher, passes it to the Connector, and launches the Connector in its own thread. The Runner then calls publisher.waitForCompletion() and blocks. In distributed mode, the Worker and Indexer processes would have been started separately beforehand. In local mode, the Runner starts the Worker and Indexer as threads in the same JVM. With this infrastructure in place, documents can begin their journey.

Step 1 — Raw data in the source system

The document begins as a row in a database table, a file on a filesystem, a message on a Kafka topic, or any other source that Lucille’s Connector knows how to read. At this point it is not yet a Lucille Document — it is raw data in whatever format the source system uses.

Step 2 — The Connector reads the record and creates a Document

Inside connector.execute(publisher), the Connector reads the next record from its source and constructs a Document with a stable, user-defined ID. The ID must be deterministic — if the pipeline is re-run, the same source record must produce the same document ID so that the Indexer’s upsert semantics correctly update rather than duplicate the record in the search backend.

The Connector calls publisher.publish(document).

Step 3 — The Publisher registers the Document and puts it on the processing queue

Before placing the document on the processing queue, the Publisher does two things in order:

Stamps the document with the run_id — an immutable field set once and used for log correlation, event routing, and Kafka topic naming throughout the document’s lifetime.
Registers the document’s ID in its accounting ledger (docIdsToTrack). Registration happens before the document is placed on the queue — the moment it is on the queue, a Worker could process it and the Indexer could send a FINISH event. If the Publisher hadn’t registered the ID yet, it would misclassify that event.

The document is then placed on the processing queue. In distributed mode, this crosses a Kafka boundary: the document becomes a message on the {pipeline}_source Kafka topic, serialized and persisted.

No lifecycle event is sent at this point — the Publisher tracks the document internally through the accounting ledger rather than via the event queue.

Step 4 — A Worker thread picks up the Document

A Worker thread is blocking on processingQueue.poll(). In distributed mode, this is a Kafka consumer poll() call — each Worker thread is an independent Kafka consumer within the consumer group, and Kafka’s consumer group protocol has assigned it one or more partitions of the source topic. The document is delivered to exactly one Worker thread.

The Worker passes the document through each Stage in the pipeline in sequence.

Step 5 — Pipeline processing

Each Stage receives the document, performs its transformation — extracting text, running NLP, generating an embedding, querying a database — and returns the result.

Step 6 — The parent Document moves to the indexing queue

Once the parent document has passed through all pipeline stages, the Worker places it on the indexing queue. In distributed mode, this crosses a second Kafka boundary: the document becomes a message on the {pipeline}_dest Kafka topic.

Step 7 — The Indexer batches and sends the Document

The Indexer thread is blocking on indexingQueue.poll(). It accumulates documents in a batch. When the batch reaches its configured size (indexer.batchSize, default 100 documents) or its timeout elapses (indexer.batchTimeout, default 100ms), the Indexer issues a single bulk API call to the search backend.

Before sending, the Indexer strips reserved internal fields (___dropped, ___skipped, ___children) and applies any configured field whitelist or blacklist. The document that reaches the search backend contains only the fields you want indexed.

The search backend accepts the document. The document now exists in the index.

Step 8 — The Indexer sends a FINISH event

After a successful bulk operation, the Indexer sends a FINISH event to the event queue for each document in the batch — including our document. In distributed mode this event travels on the {pipeline}_event_{runId} Kafka topic, which is unique per run so that events from concurrent runs never interfere with each other.

The Publisher’s polling loop receives the FINISH event, looks up the document’s ID in docIdsToTrack, and removes it. The document’s lifecycle is complete.

Step 9 — The Document is searchable

The document has been accepted by the search backend. Once the backend makes it visible — via a scheduled commit in Solr, or after the refresh interval in Elasticsearch and OpenSearch — a query can retrieve it. Lucille does not issue commits; visibility timing is controlled entirely by the search backend’s configuration. The enrichment performed during pipeline processing — the extracted text, the entity tags, the vector embedding — is available for full-text search, faceting, and semantic retrieval.

After all documents (parents and children) have completed their individual lifecycles, the framework determines that the run is complete. The Publisher continuously evaluates three conditions: (1) the Connector thread has terminated (no more documents will be published), (2) the event queue is empty (no more events are in transit), and (3) docIdsToTrack is empty (every document has reached a terminal state). All three must hold simultaneously. When they do, waitForCompletion() returns and the Runner logs the run summary.

The Detailed Path

The Happy Path above omits edge cases, error handling, logging, metrics, backpressure, routing, and child documents. This section walks through the same journey again, annotating each step with everything that actually happens. A developer debugging “why didn’t my document get indexed?” can walk through these steps to identify where the document’s journey diverged.