Architecture

Understanding Lucille’s core components, topology, and design.

Lucille separates the three concerns of ETL — reading, transforming, and writing — into distinct components that run concurrently.

Start Here

Section	What It Covers
Overview	The problem Lucille solves, the core architecture, topology, document lifecycle, and design rationale
Components	Reference pages for each component (Document, Pipeline, Publisher, Connectors, Indexers, Stages, Config)

How Components Interact

The components communicate through queues. In local mode these are in-memory LinkedBlockingQueue instances. In distributed mode they are Kafka topics. The component code is identical in both cases — only the messenger implementation changes.

Connector → [processing queue] → Worker(s) → [indexing queue] → Indexer
                                       ↓ events ↑
                                  Publisher (run accounting)

For in-depth explanations of how each subsystem works internally, see Internals.

Overview

A narrative introduction to Lucille’s architecture — the problem it solves, the core design, and how documents flow through the system.

Components

Conceptual guide to the core components of Lucille and how they work together.

Internals

In-depth explanations of how each architectural subsystem works internally and why it was designed that way.

You are viewing a snapshot of the in-development Lucille documentation.