Internals on Lucille

Messenger Abstraction

Mon, 09 Jun 2025 00:00:00 +0000

Overview

The Messenger layer is the architectural seam that makes Lucille deployment-mode independent. The same Connector, Worker, and Indexer code runs identically whether messages flow through in-memory queues, Kafka topics, or a hybrid of both. This is achieved through three interfaces and a factory pattern that selects the right implementation at runtime.

The Three Messenger Interfaces

Each Lucille component has its own messenger interface, exposing only the operations that component needs:

Message Ordering

Mon, 09 Jun 2025 00:00:00 +0000

Why Ordering Matters

Search ingestion often involves sequences of operations on the same document: a create, followed by one or more updates, possibly followed by a delete. If these operations are reordered, the final state of the search index is wrong. For example, if a sequence of create → update → delete is reordered so that the delete is processed before the update, the document survives in the index when it should have been removed. Or if the create arrives after the delete, the document reappears.

Error Handling and Fault Tolerance

Mon, 09 Jun 2025 00:00:00 +0000

Philosophy

Most search ingestion projects take an optimistic approach to error handling: ingest as much content as possible and continue past errors. A search index is often useful even if it doesn’t contain 100% of the data from the source system. Source systems routinely contain messy data where some records will inevitably cause errors in processing — a malformed PDF, a record with unexpected encoding, a field that exceeds a size limit. Stopping the entire ingest because one document out of millions is problematic would be counterproductive.

Kafka Integration

Mon, 09 Jun 2025 00:00:00 +0000

Overview

Kafka is Lucille’s distributed messaging backbone. When running in KAFKA_LOCAL or KAFKA_DISTRIBUTED mode, all inter-component communication flows through Kafka topics. This enables horizontal scaling: multiple Worker processes can consume from the same source topic, and multiple Indexer processes can consume from the same dest topic.

Topic Naming Conventions

Lucille uses four topics per pipeline, named by convention:

public static String getSourceTopicName(String pipelineName, Config config) {
 // Override: kafka.sourceTopic
 // Default: {pipelineName}_source
 return pipelineName + "_source";
}

public static String getDestTopicName(String pipelineName) {
 return pipelineName + "_dest";
}

public static String getFailTopicName(String pipelineName) {
 return pipelineName + "_fail";
}

public static String getEventTopicName(Config config, String pipelineName, String runId) {
 // Override: kafka.eventTopic
 // Default: {pipelineName}_event_{runId}
 return pipelineName + "_event_" + runId;
}

Topic	Purpose	Producers	Consumers
`{pipeline}_source`	Documents waiting to be processed	Publisher	Workers
`{pipeline}_dest`	Processed documents waiting to be indexed	Workers	Indexers
`{pipeline}_fail`	Poison-pill documents (dead letter queue)	Workers	External monitoring
`{pipeline}_event_{runId}`	Lifecycle events back to Publisher	Workers, Indexers	Publisher

The event topic is per-run (includes the runId) because each run needs its own isolated event stream. The source, dest, and fail topics are per-pipeline and persist across runs.

Metrics and Observability

Mon, 09 Jun 2025 00:00:00 +0000

Overview

Lucille uses the Codahale (Dropwizard) Metrics library for runtime observability. Every component — Publisher, Worker, Pipeline, Stage, and Indexer — reports metrics through a shared registry. These metrics are logged periodically during execution and summarized at the end of each run.

The Shared MetricRegistry

All components share a single registry:

MetricRegistry metrics = SharedMetricRegistries.getOrCreate(LogUtils.METRICS_REG);

Where LogUtils.METRICS_REG is the constant "default". Using SharedMetricRegistries means any component in the JVM can access the same registry without passing references around.