Getting Started

• 1: Why Lucille
• 2: Installation
• 3: Your First Pipeline
• 4: Running Lucille

1 - Why Lucille

What Lucille Does

Lucille is a Java-based framework for getting data into search engines and vector databases. It reads from source systems (databases, filesystems, APIs, Kafka topics), enriches the data through a configurable pipeline of processing stages, and delivers it to a search backend (Solr, OpenSearch, Elasticsearch, Pinecone, Weaviate) in batches.

The framework handles the hard parts of production search ingestion — concurrency, completion detection, fault tolerance, batching, retry, error handling, backpressure, and testability — so that you can focus on the specifics of your data and your enrichment logic.

What Lucille Excels At

Search-native indexing. Lucille’s indexers understand search engine bulk APIs, batching semantics, deletion markers, index routing, and retry logic. The document model represents data the way search engines think about it — typed fields, single-valued and multi-valued, with operations that map directly to search engine semantics. You don’t adapt a generic ETL tool to speak search; the framework already does.

Inline enrichment. A large library of built-in stages covers common search ingestion tasks without custom code: text extraction, NLP, entity recognition, language detection, embedding generation, text chunking, database lookups, HTTP enrichment, scripting (JavaScript, Python), and JSONata transformations. Enrichment happens inside the pipeline as documents flow through — not as a separate post-processing step.

Scalability without code changes. The same pipeline configuration runs in a single JVM for development and in a fully distributed Kafka deployment for production. Scaling up means adding Worker threads (local mode) or Worker processes (distributed mode). The transition is a command-line flag, not a rewrite.

Operational simplicity. Lucille is a JAR you run from the command line. It starts, processes documents, and exits. There is no server to keep running between jobs, no management UI to maintain, and no deployment infrastructure beyond the JVM.

Exact document accounting. Every document is tracked from publication through its terminal state. At the end of a run, you know exactly how many documents succeeded, failed, or were dropped — not an approximation.

Fault tolerance in distributed mode. Crash resilience via Kafka consumer group rebalancing, poison pill detection with dead-letter queues, and graceful shutdown that preserves in-flight work. One bad document doesn’t stop the run; one crashed Worker doesn’t lose data.

Configuration-driven operation. All aspects of an ingest — sources, pipelines, stages, destinations, tuning parameters — are declared in a readable HOCON configuration file. Changes are possible by editing the file alone, without rebuilding the project. Configuration is composable, parameterizable, and validatable before execution.

Testability. The entire framework runs inside a unit test with no external infrastructure. A dedicated test mode captures the complete history of every document for assertion, making it straightforward to verify pipeline correctness.

When Lucille Is the Right Choice

Lucille is a strong fit when:

• Your destination is a search engine or vector database — Solr, OpenSearch, Elasticsearch, Pinecone, or Weaviate.
• Your pipeline involves non-trivial enrichment — NLP, embeddings, OCR, database lookups, LLM-based extraction, text chunking — not just field mapping.
• You need both batch and streaming from the same pipeline — start with a batch backfill, transition to continuous streaming updates, without rewriting anything. Batch mode gives you exact accounting and structured run summaries; streaming mode gives you unbounded continuous ingestion. The same pipeline logic works in both.
• You want to start simple and scale later — develop in local mode, deploy distributed when volume demands it, without changing pipeline logic.
• You prefer code-and-config over UI-driven workflows — pipelines are version-controlled HOCON files, not visual diagrams.
• You’re a Java shop (or willing to use Java for custom components) — the framework and extension points are Java-native.

How to Know If It’s Not the Right Fit

Lucille is probably not the best choice when:

• Your destination is a data warehouse (Snowflake, BigQuery, Redshift). Lucille’s indexers target search engines and vector databases.
• You need a massive SaaS connector catalog. Lucille’s built-in connectors cover databases, filesystems, Kafka, RSS, and search engines. If you need to ingest from hundreds of SaaS applications (Salesforce, HubSpot, Zendesk), tools with larger connector ecosystems may be more practical.
• Sub-second real-time indexing latency is a hard requirement. Lucille’s streaming mode is continuous but routes through Kafka, a Worker pipeline, and a batching Indexer — adding latency that dedicated CDC-to-search connectors can avoid.
• A visual workflow editor is required. Lucille pipelines are defined in configuration files. There is no drag-and-drop UI or graphical pipeline designer.
• Your team has no Java engineers and no interest in Java. Custom connectors and stages are Java classes. The EmbeddedPython and ExternalPython stages provide a Python authoring path for enrichment logic, but the framework itself is Java.

Other Tools to Consider

If Lucille isn’t the right fit, or if you want to evaluate alternatives, tools that teams commonly consider for search-adjacent ETL include:

• Logstash — input/filter/output pipeline model, tightly integrated with the Elastic ecosystem
• OpenSearch Data Prepper — data collector oriented toward observability and log ingestion into OpenSearch
• Apache NiFi — visual dataflow tool with a large processor library and a persistent server model
• Apache Spark — distributed data processing framework suited to petabyte-scale workloads
• Airbyte — replication tool with a large SaaS connector catalog, oriented toward data warehouses
• Custom pipelines — Python scripts, Java applications, or Spark jobs grown organically over time

Each has different strengths and tradeoffs. Lucille’s niche is the intersection of search-native indexing, inline enrichment, and operational simplicity — a purpose-built tool for the specific problem of getting enriched data into search engines at scale.

2 - Installation

Do You Need to Install Lucille?

It depends on how you’re using it:

Using Lucille as a Library in an Existing Java Project

If you’re embedding Lucille inside an existing Java project (e.g., calling Runner.run() programmatically, or using Lucille’s Document API and Stages within your own application), you don’t need to install anything. Just add Lucille as a Maven dependency:

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.kmwllc</groupId>
      <artifactId>lucille-bom</artifactId>
      <version>0.9.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.kmwllc</groupId>
    <artifactId>lucille-core</artifactId>
  </dependency>
  <!-- Add plugins as needed -->
  <dependency>
    <groupId>com.kmwllc</groupId>
    <artifactId>lucille-tika</artifactId>
  </dependency>
</dependencies>

Lucille is published to Maven Central. The BOM ensures all Lucille modules are at the same version. No source build required.

Running Lucille from the Command Line (Runner, Worker, Indexer)

If you’re launching Lucille components via java -cp ... com.kmwllc.lucille.core.Runner (or Worker, WorkerIndexer, Indexer), you need the compiled JARs and their dependencies on the classpath. The easiest way to get these is to clone the repository and build from source:

git clone https://github.com/kmwtechnology/lucille.git
cd lucille
git checkout v0.9.0  # check out the release tag/branch you want to use
mvn clean install

After the build, the JARs and dependencies are in each module’s target/ directory. You can then run Lucille with:

java -Dconfig.file=my-config.conf -cp 'lucille-core/target/lucille-core-0.9.0.jar:lucille-core/target/lib/*' com.kmwllc.lucille.core.Runner

Developing New Lucille Components (Stages, Connectors, Indexers)

If you’re contributing to Lucille or developing new components that will be part of the Lucille codebase, clone the repository and build from source as described below.

Prerequisites

To build and run Lucille from source, you need:

• Java 21+ JDK (not just a JRE)
• Maven (recent version)

Java Setup (JDK 21+ Required)

Important: Before running any Lucille commands, make sure JAVA_HOME points to a JDK 21+ (not just a JRE) and that $JAVA_HOME/bin is on your PATH (or %JAVA_HOME%\bin on Windows). Maven and the java launcher rely on this.

Verify Java

java -version

You should see version 21 (or newer). If it’s missing or older than 21 install a JDK 21+ using one of the options below.

Install Options

Package manager

• macOS (Homebrew)

  brew install openjdk@21
  

• Windows (Chocolatey)

  choco install microsoft-openjdk21
  

Vendor installer

• Download a JDK 21+ installer from a vendor such as Oracle JDK.
• Run the installer, then set JAVA_HOME as shown below.

Set JAVA_HOME and PATH

macOS

export JAVA_HOME="$(/usr/libexec/java_home -v 21)"
export PATH="$JAVA_HOME/bin:$PATH"

Windows

• Open System Properties, Environment Variables.
• Create/Edit JAVA_HOME and point it to your JDK folder.
• Edit Path and add %JAVA_HOME%\bin above other Java entries.

Maven Setup

mvn -v

You should see a recent Maven version and your Java home. If mvn is not found, install Maven using one of the options below.

Install Options

Package manager

• macOS (Homebrew)

  brew install maven
  

• Windows (Chocolatey)

  choco install maven
  

Binary installer

• Download the binary zip/tar for Apache Maven from the official website.
• Add Maven’s bin/ to your PATH.

macOS

export PATH="<maven-dir>/bin:$PATH"

Windows

• Open System Properties, Environment Variables.
• Edit Path and add <maven-dir>/bin.

Clone the Repository

git clone https://github.com/kmwtechnology/lucille.git
cd lucille

To use a specific release version, check out the corresponding tag:

git checkout 0.9.0  # replace with the version you want

To see available release tags:

git tag --list

Build Lucille

mvn clean install

This compiles all modules and produces build artifacts under each module’s target/ folder. After the build:

• lucille-core/target/lucille-core-{version}.jar — the core framework JAR
• lucille-core/target/lib/ — all runtime dependencies
• lucille-plugins/lucille-tika/target/lucille-tika-{version}.jar — plugin JARs (one per plugin)
• lucille-examples/*/target/lib/ — example projects with all dependencies copied for easy classpath setup

3 - Your First Pipeline

This page walks you through creating a Lucille configuration file from scratch — a single connector reading from a CSV file, a pipeline with two stages, and an indexer writing to a local CSV output. No search backend required.

By the end, you’ll understand the structure of a Lucille config and be ready to adapt it for your own data.

Prerequisites

• Lucille built from source (see Installation)
• A terminal in the Lucille repository root

Step 1: Create a Source File

Create a file called my-source.csv with a small set of classic novels:

id,title,author,year
1,The Great Gatsby,F. Scott Fitzgerald,1925
2,To Kill a Mockingbird,Harper Lee,1960
3,1984,George Orwell,1949
4,Pride and Prejudice,Jane Austen,1813
5,The Catcher in the Rye,J.D. Salinger,1951

Step 2: Write the Config File

This config reads my-source.csv, applies two field transformations to each row, and writes the results to a new CSV file. This config has three parts:

• Connector — a FileConnector with the CSV file handler reads my-source.csv and emits each row as a Document.
• Pipeline — two stages manipulate fields on each Document: one renames title to book_title, and one adds a static source tag to every Document.
• Indexer — a CSV indexer writes the transformed Documents to an output file. No search backend is needed for this first example.

Create a file called my-first-pipeline.conf:

# ─── CONNECTORS ───────────────────────────────────────────────
# A list of connectors to run in sequence.
# Each connector reads from a source and publishes Documents.

connectors: [
  {
    # A name for this connector (appears in logs and run summary)
    name: "csv-reader"

    # The connector implementation class
    class: "com.kmwllc.lucille.connector.FileConnector"

    # Which pipeline should process this connector's documents
    pipeline: "my-pipeline"

    # Path(s) to the source file(s)
    paths: ["my-source.csv"]

    # Apply the CSV file handler (with default options) to *.csv files.
    # Each CSV file is parsed row by row; one Document is emitted per row.
    fileHandlers: {
      csv: {}
    }
  }
]

# ─── PIPELINES ────────────────────────────────────────────────
# Each pipeline is a named sequence of Stages.
# Stages transform Documents in place before they reach the Indexer.

pipelines: [
  {
    name: "my-pipeline"

    stages: [
      # Stage 1: Rename "title" to "book_title"
      {
        name: "rename-title"
        class: "com.kmwllc.lucille.stage.RenameFields"
        fieldMapping: {
          "title": "book_title"
        }
      },

      # Stage 2: Add a static field to every document
      {
        name: "add-source-tag"
        class: "com.kmwllc.lucille.stage.SetStaticFieldValue"
        fieldName: "source"
        fieldValue: "my-first-pipeline"
      }
    ]
  }
]

# ─── INDEXER ──────────────────────────────────────────────────
# The Indexer sends processed Documents to a destination.
# We use the CSV indexer here so no search backend is needed.

indexer {
  type: "CSV"
}

# CSV indexer settings
csv {
  # Which fields to include in the output (must match field names after pipeline processing)
  columns: ["id", "book_title", "author", "year", "source"]

  # Where to write the output
  path: "my-output.csv"

  # Include a header row
  includeHeader: true
}

Step 3: Run It

java -Dconfig.file=my-first-pipeline.conf \
  -cp 'lucille-core/target/lucille-core-*.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner

Step 4: Check the Output

Open my-output.csv:

id,book_title,author,year,source
1,The Great Gatsby,F. Scott Fitzgerald,1925,my-first-pipeline
2,To Kill a Mockingbird,Harper Lee,1960,my-first-pipeline
3,1984,George Orwell,1949,my-first-pipeline
4,Pride and Prejudice,Jane Austen,1813,my-first-pipeline
5,The Catcher in the Rye,J.D. Salinger,1951,my-first-pipeline

The title field was renamed to book_title, and a source field was added to every document.

Step 5: Check the Logs

You should see output like:

INFO  Runner: Pipeline Configuration is valid.
INFO  Runner: Connector Configuration is valid.
INFO  Runner: Indexer Configuration is valid.
INFO  Runner: Starting run with id ...
INFO  Runner: Running connector csv-reader feeding to pipeline my-pipeline
INFO  PublisherImpl: First doc published after ... ms
INFO  WorkerPool: 5 docs processed. One minute rate: ... docs/sec. Mean pipeline latency: ... ms/doc.
INFO  Stage: Stage rename-title metrics. Docs processed: 5. Mean latency: ... ms/doc. Children: 0. Errors: 0.
INFO  Stage: Stage add-source-tag metrics. Docs processed: 5. Mean latency: ... ms/doc. Children: 0. Errors: 0.
INFO  Runner: RUN SUMMARY: Success. 1/1 connectors complete. All published docs succeeded.
INFO  Runner: Run took ... secs.

Understanding the Config Structure

Every Lucille config has three required parts:

Each connector specifies:

• name — for logging and the run summary
• class — the fully qualified Java class that implements the connector
• pipeline — which pipeline processes this connector’s output
• Connector-specific settings (e.g., paths for FileConnector)

Each stage in a pipeline specifies:

• class — the fully qualified Java class that implements the stage
• name (optional) — for logging and per-stage metrics
• Stage-specific parameters (e.g., fieldMapping for RenameFields)

The indexer specifies:

• type — a shorthand for built-in indexers (Solr, OpenSearch, Elasticsearch, CSV)
• Or class — for plugin or custom indexers
• Backend-specific settings in a separate block (e.g., csv {}, solr {}, opensearch {})

What Have We Accomplished?

At first glance, the config above might feel disproportionate to the task. We renamed one field and added a static value — a transformation you could express in a single line of shell:

sed '1s/title/book_title/' my-source.csv \
  | awk -F, 'BEGIN{OFS=","} NR==1{print $0,"source"} NR>1{print $0,"my-first-pipeline"}' \
  > my-output.csv

That’s fair. For this specific transformation on this specific file, a shell one-liner wins. The value of Lucille comes from what you can do next — and how little you have to change to get there.

1. Index to any search backend

Replace the CSV indexer with a real search backend — OpenSearch, Elasticsearch, or Solr — and your trivial file experiment becomes a real search ingestion pipeline. For OpenSearch:

indexer {
  type: "OpenSearch"
}

opensearch {
  url: ${?OPENSEARCH_URL}
  index: "my-books"
}

That’s the only change. The connector, the pipeline, the stages — everything else stays exactly the same. If you later want to switch from OpenSearch to Solr, you change the indexer block and add a solr {} block. No code, no build cycle, no rewrite.

2. Iterate on your pipeline

Add stages, chain transformations, and introduce conditions — all in config. Want to generate an embedding for each book title and store it as a vector field? Add a stage. Want to skip that stage for documents that already have an embedding? Add a condition. Browse the Stage Reference for the full library of available transformations.

None of this requires a build. Edit the config file, rerun Lucille, see the results. The pipeline is the config.

3. Scale

In our example, five rows processed in milliseconds and scale was not a concern. But now imagine the CSV has a million rows, each containing an S3 URL to a large PDF. Your pipeline must fetch each PDF, extract text with Tika, run OCR on scanned pages, and generate embeddings. That pipeline takes hours or days on a single thread.

Step 1: add worker threads. Lucille’s local mode uses a thread pool for document processing. Increase the number of threads to parallelize across available CPU cores and overlap IO:

worker {
  threads: 8
}

No other changes needed. The same config, more threads.

Step 2: distribute across machines. If one machine isn’t enough, switch to distributed mode. Lucille’s components — Runner, Workers, and Indexer — run as separate JVM processes and communicate through Kafka. Here’s what changes:

• The Runner gets the -usekafka flag: com.kmwllc.lucille.core.Runner -usekafka
• Workers and the Indexer are started as separate processes on separate machines (or in separate Kubernetes pods): com.kmwllc.lucille.core.Worker my-pipeline and com.kmwllc.lucille.core.Indexer my-pipeline
• A kafka {} block with bootstrapServers is added to the config

Your connector definition and your pipeline stages are untouched. See Production Deployment for the full distributed mode setup.

At first, with a trivial five-row example that ran in under a second, Lucille looked like overkill compared to a shell script. But as soon as we wanted to ingest into a real search backend, iterate on enrichment logic, and scale out to handle large volumes of heavy documents, we could do all of that in a config-driven way — switching backends, adding stages, and distributing across machines — without having to solve any of those problems from scratch.

Consider what Lucille is doing on our behalf in the scaled-out version of this pipeline. The Runner is reading our million-row CSV and producing each row as a Document onto a Kafka topic, applying backpressure so it doesn’t publish faster than Workers can consume. Multiple Worker processes — running on different machines or in different Kubernetes pods — are each pulling Documents off that topic in parallel, each running multiple threads, each thread independently fetching a PDF from S3, extracting text, running OCR, and generating embeddings. As each Document finishes processing, a lifecycle event is sent back to the Runner so it can track exactly how many Documents have succeeded, failed, or been dropped. If a Worker crashes mid-run, Kafka’s consumer group protocol detects it and reassigns its unacknowledged Documents to another Worker — nothing is silently lost. If a single malformed PDF repeatedly crashes a Worker, Lucille routes it to a dead-letter topic after a configurable number of retries and keeps the rest of the run going. The Indexer is collecting processed Documents and sending them to the search backend in tunable batches, managing bulk API semantics, and handling retries on transient backend errors. Throughout the run, Lucille is logging throughput metrics, per-stage latency, and document counts at regular intervals. When the last Document reaches a terminal state, Lucille determines that the run is complete and prints a structured summary — how many succeeded, how many failed, how many were dropped, and how long it took.

All of that happens without any code on our part. Our contribution is the config file we wrote in Step 2.

4 - Running Lucille

Try the Examples

Lucille’s lucille-examples module contains several runnable examples:

To run the simple CSV example, start an instance of Apache Solr on port 8983 and create a collection called quickstart. Then from lucille-examples/lucille-simple-csv-solr-example, run:

mvn clean install
./scripts/run_ingest.sh

This executes Lucille with simple-csv-solr-example.conf, which reads a CSV of top songs and sends each row as a document to Solr. After the run, issue a commit with openSearcher=true on your quickstart collection and run a *:* query in the Solr admin dashboard to see the indexed documents.

Local Mode

Local mode runs all Lucille components — Connector, Workers, and Indexer — as threads inside a single JVM. No Kafka or external messaging is required. This is the right mode for development, testing, and single-machine production runs.

Prepare a Configuration File

Point Lucille at a config file declaring your connectors, pipelines, and indexer. See Configuration Management for the full schema, or follow Your First Pipeline for a step-by-step walkthrough.

Run the Runner

From the repository root:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner

See the troubleshooting guide if Lucille doesn’t start as expected.

Distributed Mode

Distributed mode runs each Lucille component as its own JVM process, communicating through Apache Kafka. Use this when you need to scale out across multiple machines or processes.

This guide assumes Kafka and your destination system are already running and reachable. For production deployment patterns including Docker Compose and Kubernetes, see Production Deployment.

Prepare a Configuration File

Use a single config file shared by all components. It must declare your connector(s), pipeline(s), kafka configuration, and your indexer with its backend config.

Start the Runner

The Runner publishes documents to the Kafka source topic, listens for pipeline run events, logs run statistics, and waits for the run to complete:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner \
  -usekafka

Start Workers

Each Worker consumes documents from the Kafka source topic, processes each document through the configured pipeline, and writes processed documents to the Kafka destination topic:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Worker \
  <pipeline-name>

Start Workers before the Runner to avoid Kafka consumer group rebalancing delays while documents are already in flight.

Start the Indexer

The Indexer consumes documents from the Kafka destination topic and sends batches to the configured search backend:

java \
  -Dconfig.file=<PATH/TO/YOUR/CONFIG.conf> \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Indexer \
  <pipeline-name>

See the troubleshooting guide if components don’t connect as expected.

Verifying Your Run

During the Run

Lucille logs throughput and latency metrics periodically:

25/10/31 13:40:21 INFO WorkerPool: 27017 docs processed. One minute rate: 1787.10 docs/sec. Mean pipeline latency: 10.63 ms/doc.
25/10/31 13:40:22 INFO PublisherImpl: 37029 docs published. One minute rate: 3225.69 docs/sec. Waiting on 21014 docs.
25/10/31 13:40:22 INFO Indexer: 17016 docs indexed. One minute rate: 455.07 docs/sec. Mean backend latency: 6.90 ms/doc.

At Completion

At the end of every run, Lucille prints a stage-by-stage performance summary followed by a final run result:

25/10/31 13:46:47 INFO Stage: Stage rename-title metrics. Docs processed: 200000. Mean latency: 0.0003 ms/doc. Children: 0. Errors: 0.
25/10/31 13:46:47 INFO Stage: Stage add-source-tag metrics. Docs processed: 200000. Mean latency: 0.3532 ms/doc. Children: 0. Errors: 0.
25/10/31 13:46:47 INFO Runner:
RUN SUMMARY: Success. 1/1 connectors complete. All published docs succeeded.
connector1: complete. 200000 docs succeeded. 0 docs failed. 0 docs dropped. Time: 416.47 secs.
25/10/31 13:46:47 INFO Runner: Run took 417.46 secs.

Check Your Search Backend

After the run, verify that documents are visible in your target system. For Solr, a commit with openSearcher=true is required before documents appear in query results — Lucille does not issue a commit automatically. For Elasticsearch and OpenSearch, documents are available after the index refresh interval (default: 1 second).