Getting Started on Lucille

Why Lucille

Mon, 09 Jun 2025 00:00:00 +0000

What Lucille Does

Lucille is a Java-based framework for getting data into search engines and vector databases. It reads from source systems (databases, filesystems, APIs, Kafka topics), enriches the data through a configurable pipeline of processing stages, and delivers it to a search backend (Solr, OpenSearch, Elasticsearch, Pinecone, Weaviate) in batches.

The framework handles the hard parts of production search ingestion — concurrency, completion detection, fault tolerance, batching, retry, error handling, backpressure, and testability — so that you can focus on the specifics of your data and your enrichment logic.

Installation

Fri, 31 Oct 2025 00:00:00 +0000

Do You Need to Install Lucille?

It depends on how you’re using it:

Using Lucille as a Library in an Existing Java Project

If you’re embedding Lucille inside an existing Java project (e.g., calling Runner.run() programmatically, or using Lucille’s Document API and Stages within your own application), you don’t need to install anything. Just add Lucille as a Maven dependency:

<dependencyManagement>
 <dependencies>
 <dependency>
 <groupId>com.kmwllc</groupId>
 <artifactId>lucille-bom</artifactId>
 <version>0.9.0</version>
 <type>pom</type>
 <scope>import</scope>
 </dependency>
 </dependencies>
</dependencyManagement>

<dependencies>
 <dependency>
 <groupId>com.kmwllc</groupId>
 <artifactId>lucille-core</artifactId>
 </dependency>
 <!-- Add plugins as needed -->
 <dependency>
 <groupId>com.kmwllc</groupId>
 <artifactId>lucille-tika</artifactId>
 </dependency>
</dependencies>

Lucille is published to Maven Central. The BOM ensures all Lucille modules are at the same version. No source build required.

Your First Pipeline

Mon, 09 Jun 2025 00:00:00 +0000

This page walks you through creating a Lucille configuration file from scratch — a single connector reading from a CSV file, a pipeline with two stages, and an indexer writing to a local CSV output. No search backend required.

By the end, you’ll understand the structure of a Lucille config and be ready to adapt it for your own data.

Prerequisites

Lucille built from source (see Installation)
A terminal in the Lucille repository root

Step 1: Create a Source File

Create a file called my-source.csv with a small set of classic novels:

Running Lucille

Mon, 09 Jun 2025 00:00:00 +0000

Try the Examples

Lucille’s lucille-examples module contains several runnable examples:

Example	Description
lucille-simple-csv-solr-example	Ingest a CSV file into Solr in local mode. A good first example.
lucille-distributed-example	Full distributed mode with Runner, Worker, Indexer, Kafka, ZooKeeper, and Solr each running in their own Docker container via Docker Compose. Includes an integration test that verifies the ingest result. Run with `mvn verify -Pnightly` from that directory.

To run the simple CSV example, start an instance of Apache Solr on port 8983 and create a collection called quickstart. Then from lucille-examples/lucille-simple-csv-solr-example, run: