Overview on Lucille

The Problem of Search ETL

Mon, 09 Jun 2025 00:00:00 +0000

Organizations build search applications when their users need to find information across large, heterogeneous collections of data — product catalogs, legal documents, support tickets, research papers, internal wikis, file shares. The data lives in databases, cloud storage, APIs, and content management systems, often in different formats and with varying levels of structure. Before any of it becomes searchable, it must be extracted from these sources, cleaned, enriched (with metadata, classifications, embeddings, or extracted entities), and delivered to a search engine or vector database in the right format. This process — getting source data into a search-ready state — is search ETL.

Parallelizing Search ETL

Mon, 09 Jun 2025 00:00:00 +0000

Lucille addresses all of the pain points described in The Problem of Search ETL — error handling, observability, pipeline composition, fan-out, configuration management, and scalability. This page focuses on the last and most fundamental of those problems: the sequential loop can’t go faster.

Lucille’s answer is a parallel architecture whose components derive directly from the structure of the problem. Each of the three functions of search ETL — acquiring data, enriching it, and indexing it — becomes its own independent concurrent component, operating at its own pace, limited only by its own resources rather than by the speed of the others.

From Single Process to Distributed

Mon, 09 Jun 2025 00:00:00 +0000

The previous page showed how to parallelize search ingestion using three concurrent components — Connector, Workers, and Indexers — communicating through a processing queue and an indexing queue. What we have not yet discussed is what those queues actually are.

What a Queue Must Provide

The queues in Lucille’s architecture have a specific set of requirements. They must support concurrent access from multiple producers and consumers. They must support a blocking poll() operation so that a Worker or Indexer that finds an empty queue waits efficiently rather than spinning. And when multiple Workers are calling poll() concurrently, work must be distributed fairly among them — each document should go to exactly one Worker.

Document Lifecycle

Mon, 09 Jun 2025 00:00:00 +0000

This page traces a single Document from the moment it exists only as raw data in a source system to the moment it is retrievable by a search query. It takes the Lucille architecture as a given — for the conceptual explanation of how the system is structured, see Parallelizing Search ETL and Pluggable Queueing.

The Happy Path

Before a document’s lifecycle begins, the system components must be set up. A batch ingest is launched via the Runner, which first validates the configuration. Assuming a single-connector config, the Runner creates a Publisher, passes it to the Connector, and launches the Connector in its own thread. The Runner then calls publisher.waitForCompletion() and blocks. In distributed mode, the Worker and Indexer processes would have been started separately beforehand. In local mode, the Runner starts the Worker and Indexer as threads in the same JVM. With this infrastructure in place, documents can begin their journey.

Design Rationale

Mon, 09 Jun 2025 00:00:00 +0000

These are the overarching design decisions that shape everything about Lucille. Every architectural choice, every API design, and every operational feature must be consistent with these principles. They emerged from years of building search ingestion frameworks for customer projects and were refined through production deployments.

Part I: Guiding Principles

1. Built for search

The system should be purpose-built for getting data into search engines and vector databases — not a general-purpose ETL tool adapted for search. This focus should shape the document model, the identity model, the wire format, and the operations the system supports natively.