Overview
A narrative introduction to Lucille’s architecture — the problem it solves, the core design, and how documents flow through the system.
Architecture
Understanding Lucille’s core components, topology, and design.
Lucille separates the three concerns of ETL — reading, transforming, and writing — into distinct components that run concurrently.
Start Here
| Section | What It Covers |
|---|---|
| Overview | The problem Lucille solves, the core architecture, topology, document lifecycle, and design rationale |
| Components | Reference pages for each component (Document, Pipeline, Publisher, Connectors, Indexers, Stages, Config) |
How Components Interact
The components communicate through queues. In local mode these are in-memory LinkedBlockingQueue instances. In distributed mode they are Kafka topics. The component code is identical in both cases — only the messenger implementation changes.
Connector → [processing queue] → Worker(s) → [indexing queue] → Indexer
↓ events ↑
Publisher (run accounting)
For in-depth explanations of how each subsystem works internally, see Internals.
Components
Conceptual guide to the core components of Lucille and how they work together.
Internals
In-depth explanations of how each architectural subsystem works internally and why it was designed that way.