The Problem of Search ETL
Why a simple sequential loop falls short for production search ingestion, and how Lucille addresses the pitfalls.
Overview
A narrative introduction to Lucille’s architecture — the problem it solves, the core design, and how documents flow through the system.
Read these pages in order for a complete introduction to Lucille’s architectural design.
- The Problem of Search ETL — Why a simple sequential ingest loop falls short in production and what needs to change.
- Parallelizing Search ETL — How Lucille maps the three ETL functions to concurrent components communicating through queues, and how it tracks document lifecycle across an asynchronous system.
- From Single Process to Distributed — How the same pipeline code runs identically in a single JVM or distributed across machines, and how the Messenger abstraction makes this possible.
- Topology — Batch and streaming models, the WorkerIndexer, and deployment configurations from single-JVM to fully distributed.
- Document Lifecycle — The complete journey of a single Document through the system, plus cross-cutting concerns like logging, metrics, and testing.
- Design Rationale — The 24 requirements that govern Lucille’s architecture and why the system is designed the way it is.
Parallelizing Search ETL
How Lucille structures search ingestion as three concurrent components communicating through queues, and how it tracks document lifecycle across an asynchronous system.
From Single Process to Distributed
How the same pipeline code runs identically in a single JVM or distributed across machines — and how the Messenger abstraction makes this possible.
Topology
Lucille can be configured to best support your use case.
Document Lifecycle
The complete journey of a single Document through Lucille, from raw data in a source system to a searchable record in the search backend.
Design Rationale
The guiding principles that govern Lucille’s architecture, and the features that achieve them.