Getting Started
What Lucille is, how it works, and how to run your first pipeline.
Documentation pre-release
Work in Progress. This documentation is being actively developed by the Lucille maintainers with AI assistance. Our goal is for every page to be fully shaped by human input, requirements, and feedback. While many pages have undergone extensive human review, others have not and may contain inaccuracies. We’re releasing early to get feedback that helps us improve. Let us know if something looks wrong by submitting a PR using the “edit this page” feature or emailing us at info@kmwllc.com.
Lucille is a Java-based search ETL framework that reads data from sources, enriches it through configurable pipelines, and delivers it to search engines and vector databases.
Find Your Section
| Section | Who It’s For | What You’ll Find |
|---|---|---|
| Getting Started | Anyone new to Lucille | Installation, first pipeline tutorial, and why Lucille exists. |
| Architecture | Anyone who wants to understand why the system is designed the way it is | Design motivations, guiding principles, component roles and interactions, and deep dives into internal mechanisms. The aim is conceptual understanding. |
| Ingest Designer Guide | Someone writing Lucille configs to define ingests — no Java required | How to write a config, pipeline definition patterns, the full catalogue of connectors/stages/indexers, conditions reference, and task-oriented cookbooks. |
| Component Developer Guide | Java developers implementing new Connectors, Stages, or Indexers | What the framework provides vs. what you implement, skeleton code, the SPEC validation system, testing patterns, and API quick reference. |
| Operations Guide | Someone deploying and running Lucille in dev, staging, or production | Deployment modes (local, distributed, streaming, hybrid), configuration management, performance tuning, logging, security, troubleshooting, and the support matrix. |
| Contributor Guide | Someone making a PR against the Lucille codebase | Project structure, coding conventions, setup standards, and how the project is maintained. |
Also available: FAQ, Glossary, Releases.
For Documentation Contributors
Each section targets a specific reader with a specific goal. When adding or moving content, use this principle:
- Architecture answers “why” and “how it works” — design decisions, rationale, component interactions. No config syntax tutorials, no step-by-step instructions.
- Ingest Designer Guide answers “how do I configure this” — practical reference for someone assembling an ingest from existing components. Config examples, parameter tables, gotchas, cookbooks.
- Component Developer Guide answers “how do I build this” — guidance for someone writing Java code against Lucille’s interfaces. Code skeletons, API patterns, testing infrastructure.
- Operations Guide answers “how do I deploy and run this” — deployment modes, environment setup, monitoring, troubleshooting. Operational patterns and production concerns.
- Contributor Guide answers “how do I contribute to the project” — coding standards, project structure, PR process.
If content serves multiple audiences, place it where the primary audience would look and add a cross-link from the secondary location.
Architecture
Understanding Lucille’s core components, topology, and design.
Ingest Designer Guide
Everything you need to design and configure Lucille ingests — from writing your first config to advanced patterns.
Component Developer Guide
Practical guidance for developers implementing new Connectors, Stages, and Indexers for Lucille.
Operations Guide
Configuration, deployment, monitoring, performance, security, and troubleshooting for Lucille.
Contributor Guide
For contributors to the Lucille codebase — project structure, setup, and coding standards.
Concepts and Glossary
Definitions of key terms and concepts used throughout Lucille’s documentation and codebase.
FAQ
Answers to common questions about running, configuring, and extending Lucille.