Deployment on Lucille

Local Batch

Mon, 09 Jun 2025 00:00:00 +0000

Local Mode

In local mode, you start a single Java process at the command line, invoking the main() method of com.kmwllc.lucille.core.Runner. Alternatively, if you are using Lucille as a dependency in your own Java project, you call Runner.run() from your code.

All components (Connector, Workers, Indexer) run as threads in a single JVM. Communication uses in-memory queues. There is no external infrastructure required — no Kafka, no ZooKeeper, no separate processes.

Distributed Batch

Mon, 09 Jun 2025 00:00:00 +0000

Kafka-Distributed Mode

In distributed mode, each Lucille component runs as its own JVM process. All inter-component communication uses Kafka topics.

Best for: Large-scale batch ingests, multi-machine deployments, horizontal scaling.

Prerequisites

Before starting any Lucille component in distributed mode, ensure:

Kafka is running and reachable from all machines that will run Lucille components
Kafka topics are created for each pipeline referenced by your connectors (or auto-creation is enabled on the broker) — see Kafka Topic Setup for details
Workers and Indexer are running for each pipeline referenced by your connectors — see Starting Each Component
Search backend collection/index exists (Solr, OpenSearch, Elasticsearch, etc.)
ZooKeeper is running (only if using worker.maxRetries for poison-pill protection)
Same config file is available to all Lucille processes (Runner, Workers, Indexer)

Start Order

Start components in this order:

Distributed Streaming

Mon, 09 Jun 2025 00:00:00 +0000

Streaming Mode (Connectorless)

In streaming mode, Lucille runs without a Runner or Connector. An external system places documents onto a Kafka source topic directly, and one or more Worker or WorkerIndexer processes consume and process them continuously — with no run boundary or completion accounting.

When to use: Keeping a search index in sync with a live system (e.g., consuming from a CDC topic, Debezium, or any Kafka producer). The pipeline logic is identical to batch mode; only the execution model changes.

Hybrid Streaming

Mon, 09 Jun 2025 00:00:00 +0000

Hybrid Streaming Mode

In hybrid streaming mode, Lucille runs one or more WorkerIndexer processes that co-locate document processing and indexing in a single JVM. Like distributed streaming, there is no Runner or Connector — an external system places documents onto a Kafka source topic, and the WorkerIndexer processes consume, enrich, and index them continuously.

When to use: Streaming ingestion where you want the horizontal scalability of distributed mode but with simpler operations — fewer processes to manage, no separate Indexer deployment, and reduced network hops between processing and indexing.

Docker Compose

Mon, 09 Jun 2025 00:00:00 +0000

Docker Compose Deployment

Docker Compose is the simplest way to run Lucille in distributed mode. The lucille-distributed-example provides a complete working template. The key structure:

services:
 zookeeper:
 # Bundled with Kafka; required for Kafka's internal coordination
 image: ...

 kafka:
 depends_on: [zookeeper]
 healthcheck:
 test: ["CMD", "kafka-topics.sh", "--bootstrap-server=kafka:9092", "--list"]

 worker:
 depends_on:
 kafka: { condition: service_healthy }
 entrypoint: java -Dconfig.file=/conf/config.conf -cp '/target/lib/*'
 com.kmwllc.lucille.core.Worker my-pipeline

 indexer:
 depends_on:
 kafka: { condition: service_healthy }
 solr: { condition: service_healthy }
 healthcheck:
 # Delegate healthcheck to the search backend — Runner waits for this before starting
 test: curl -f 'http://solr:8983/solr/...'
 entrypoint: |
 # Create the collection, then start the Indexer
 curl 'http://solr:8983/solr/admin/collections?action=CREATE&name=quickstart...'
 java -Dconfig.file=/conf/config.conf -cp '/target/lib/*'
 com.kmwllc.lucille.core.Indexer my-pipeline

 runner:
 depends_on:
 kafka: { condition: service_healthy }
 indexer: { condition: service_healthy }  # Ensures Indexer + backend are ready
 entrypoint: java -Dconfig.file=/conf/config.conf -cp '/target/lib/*'
 com.kmwllc.lucille.core.Runner -usekafka

Key design points from the example:

Kubernetes

Mon, 09 Jun 2025 00:00:00 +0000

Kubernetes Deployment

Lucille’s architecture maps naturally onto Kubernetes primitives.

Batch Jobs as Kubernetes CronJobs

For scheduled batch ingests, package Lucille as a container and run it as a CronJob. When the run completes, the container exits — there is no long-running process to manage between runs.

Minimal Dockerfile:

FROM eclipse-temurin:17-jre

WORKDIR /app

# Copy all JARs (lucille-core + dependencies) from the Maven build output
COPY target/lib/ lib/

# Copy your pipeline configuration
COPY conf/ conf/

# The config file path is provided via an environment variable at runtime
ENV CONF=""
ENTRYPOINT ["sh", "-c", "java -Xmx4g -Dconfig.file=${CONF} -cp 'lib/*' com.kmwllc.lucille.core.Runner"]

Build the image after running mvn clean install in your project (which copies all dependencies to target/lib/):

Production Operations

Mon, 09 Jun 2025 00:00:00 +0000

Memory Sizing

Rule of thumb: worker.threads × (Stage resource usage). A pipeline that loads a 500MB NLP model uses 500MB × number of Worker threads.

Component	Typical Heap Size
Runner (local, lightweight pipeline)	512MB–2GB
Runner (local, ML stages)	4GB–16GB per stage model × threads
Worker (distributed, no ML)	512MB–1GB
Worker (distributed, with ML models)	2GB–8GB per model
Indexer	512MB–1GB

Always set -Xmx explicitly. Avoid letting Java use all available RAM.