Deployment

• 1: Local Batch
• 2: Distributed Batch
• 3: Distributed Streaming
• 4: Hybrid Streaming
• 5: Docker Compose
• 6: Kubernetes
• 7: Production Operations

1 - Local Batch

Local Mode

In local mode, you start a single Java process at the command line, invoking the main() method of com.kmwllc.lucille.core.Runner. Alternatively, if you are using Lucille as a dependency in your own Java project, you call Runner.run() from your code.

All components (Connector, Workers, Indexer) run as threads in a single JVM. Communication uses in-memory queues. There is no external infrastructure required — no Kafka, no ZooKeeper, no separate processes.

Best for: Development, scheduled batch jobs, single-machine production runs.

Note: While local mode is the first mode you’d want to use when developing a pipeline or doing a proof-of-concept, it is not at all a “toy” mode. It is fully suitable for production batch workflows assuming you don’t need the additional parallelism and crash resilience features of distributed mode.

Prerequisites

To run Lucille in local mode you need:

1. A Lucille configuration file — specifies connectors, pipelines, stages, and indexer settings. See Configuration for details.
2. The Lucille JAR and lib directory — the compiled JAR plus its dependency libraries, both placed on the classpath.

Running from the Command Line

A typical invocation looks like:

java \
  -Xmx4g \
  -Dconfig.file=/path/to/application.conf \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner

Running Programmatically

If Lucille is a dependency in your own project, you can trigger a run from Java code:

Config config = ConfigFactory.load("application.conf");
Runner.run(config);

Scaling in Local Mode

In local mode, the main way to scale is to increase the number of Worker threads. This allows you to:

• Do more I/O concurrently (if the pipeline is I/O-bound)
• Take advantage of available CPU cores for pipeline processing

Each Worker thread runs an independent copy of the pipeline, and the Worker threads together process documents concurrently.

worker {
  threads: 8
}

2 - Distributed Batch

Kafka-Distributed Mode

In distributed mode, each Lucille component runs as its own JVM process. All inter-component communication uses Kafka topics.

Best for: Large-scale batch ingests, multi-machine deployments, horizontal scaling.

Prerequisites

Before starting any Lucille component in distributed mode, ensure:

• Kafka is running and reachable from all machines that will run Lucille components
• Kafka topics are created for each pipeline referenced by your connectors (or auto-creation is enabled on the broker) — see Kafka Topic Setup for details
• Workers and Indexer are running for each pipeline referenced by your connectors — see Starting Each Component
• Search backend collection/index exists (Solr, OpenSearch, Elasticsearch, etc.)
• ZooKeeper is running (only if using worker.maxRetries for poison-pill protection)
• Same config file is available to all Lucille processes (Runner, Workers, Indexer)

Start Order

Start components in this order:

1. ZooKeeper (if using worker.maxRetries)
2. Kafka
3. Workers — start consuming before any documents arrive
4. Indexer — create the search backend collection/index first, then start the Indexer process
5. Runner — triggers the run once Workers and Indexer are ready

The Runner should be started last. It immediately begins publishing documents to Kafka. If Workers or the Indexer aren’t ready, documents will queue on the source topic (which is fine) but the run won’t complete until they start consuming.

Kafka Configuration

All distributed-mode Kafka settings belong in a top-level kafka {} block shared by all components:

kafka {
  bootstrapServers: "kafka1:9092,kafka2:9092"

  # Consumer group ID for all Lucille Workers and Indexers
  consumerGroupId: "lucille_workers"

  # Maximum time between polls before consumer is evicted from consumer group
  maxPollIntervalSecs: 600

  # Maximum request size in bytes — increase for large documents
  maxRequestSize: 250000000

  # TLS/SASL — provide Java properties files for detailed Kafka client config
  securityProtocol: "SSL"
  consumerPropertyFile: "/path/to/consumer.properties"
  producerPropertyFile: "/path/to/producer.properties"
  adminPropertyFile: "/path/to/admin.properties"

  # Custom document serializer/deserializer (override only if needed)
  documentDeserializer: "com.kmwllc.lucille.message.KafkaDocumentDeserializer"
  documentSerializer: "com.kmwllc.lucille.message.KafkaDocumentSerializer"

  # Set to false to disable event tracking (FINISH/FAIL/DROP events won't be published)
  events: true

  # Override source topic name (default: auto-generated from pipeline name)
  sourceTopic: "my-pipeline-source"

  # CAUTION: Setting eventTopic disables per-run topic isolation. Only use this
  # in streaming/connectorless mode where no Runner is coordinating the run.
  # In batch mode with a Runner, omit this — Lucille generates a per-run event topic
  # automatically to prevent cross-run event interference.
  # eventTopic: "lucille_events"
}

Kafka Topic Setup

Lucille uses Kafka topics to pass documents between components. If your Kafka broker has auto.create.topics.enable set to true (the default), you don’t need to create any topics yourself. If auto-create is disabled, you must pre-create certain topics before starting Lucille.

If auto-create is enabled

Lucille will create topics automatically on first use. No manual setup is required. However:

• Auto-created topics use the broker’s num.partitions default (typically 1). This limits parallelism — see the partition guidance below.
• Lucille does not clean up topics after a run. Event topics accumulate over time (one per run). Consider a retention policy or periodic cleanup.
• The event topic is an exception: Lucille always creates it explicitly via the Kafka Admin API with exactly 1 partition, regardless of auto.create.topics.enable. This requires Admin API permissions on the Kafka cluster.

If auto-create is disabled

Create the following topics before starting Lucille:

Example — for a pipeline named my-pipeline with 8 Worker threads:

kafka-topics.sh --create --topic my-pipeline_source \
  --partitions 8 --replication-factor 2 \
  --bootstrap-server kafka:9092

kafka-topics.sh --create --topic my-pipeline_dest \
  --partitions 1 --replication-factor 2 \
  --bootstrap-server kafka:9092

# Only if worker.maxRetries is configured:
kafka-topics.sh --create --topic my-pipeline_fail \
  --partitions 1 --replication-factor 2 \
  --bootstrap-server kafka:9092

You do not need to create the event topic — Lucille creates it automatically at the start of each run via the Admin API. This requires that the Kafka user Lucille connects with has CreateTopics permission. If your Kafka cluster requires separate admin credentials, provide them via kafka.adminPropertyFile.

Topic details

Source topic ({pipeline}_source)

The Runner publishes documents here; Workers consume from it. The partition count determines the maximum number of Worker threads that can consume in parallel — excess threads beyond the partition count will sit idle. Set the partition count to at least the total number of Worker threads you plan to run across all Worker processes.

Override the name with kafka.sourceTopic in your config if you need a custom topic name (e.g., when consuming from a topic managed by another system in streaming mode).

Dest topic ({pipeline}_dest)

Workers publish processed documents here; the Indexer consumes from it. A single partition is sufficient for most deployments because the Indexer is rarely the bottleneck — it sends batched bulk requests to the search backend and is I/O-bound on the backend’s write capacity, not on Kafka consumption. Multiple partitions are only needed if you run multiple Indexer processes (uncommon).

The name is not overridable — it is always {pipeline}_dest.

Event topic ({pipeline}_event_{runId})

Workers and Indexers publish lifecycle events here (CREATE, FINISH, FAIL, DROP); the Publisher on the Runner process consumes them to track run completion. This topic must have exactly 1 partition to guarantee FIFO ordering — if events arrive out of order, the Publisher’s accounting logic can be corrupted (e.g., a child’s FINISH arriving before its CREATE).

Lucille creates this topic explicitly via the Kafka Admin API at the start of each batch run. In batch mode, each run gets its own event topic (the run ID is in the topic name), which prevents events from one run interfering with another when multiple runs overlap. You do not pre-create it.

In streaming mode (no Runner), you can either disable events entirely (kafka.events: false) or set a fixed topic name (kafka.eventTopic: "my_fixed_event_topic") and pre-create it with 1 partition.

Fail topic ({pipeline}_fail)

The Worker publishes poison-pill documents here when they exceed worker.maxRetries. This topic is only used when worker.maxRetries is configured (which also requires ZooKeeper). If you don’t use retry tracking, this topic is never written to and doesn’t need to exist.

Documents in the fail topic can be inspected with standard Kafka tooling, corrected if needed, and replayed by pointing a connector at the topic. The topic relies on auto-creation or must be pre-created by an admin — Lucille does not create it via the Admin API.

The name is not overridable — it is always {pipeline}_fail.

Permissions required

The Kafka user Lucille connects with needs:

Starting Each Component

Runner

The Runner publishes documents to Kafka and waits for all work to complete:

java \
  -Dconfig.file=/path/to/config.conf \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Runner \
  -usekafka

Workers

Start one or more Worker processes, each consuming from the Kafka source topic:

java \
  -Dconfig.file=/path/to/config.conf \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Worker \
  my-pipeline-name

Adding more Worker processes increases throughput proportionally. Workers join the same Kafka consumer group — Kafka distributes partitions automatically.

Indexer

java \
  -Dconfig.file=/path/to/config.conf \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Indexer \
  my-pipeline-name

If the search backend collection does not yet exist, create it before launching the Indexer:

# Create Solr collection, then start Indexer
curl 'http://solr:8983/solr/admin/collections?action=CREATE&name=my-collection&numShards=1&collection.configName=_default'
java -Dconfig.file=/conf/config.conf -cp '/target/lib/*' com.kmwllc.lucille.core.Indexer my-pipeline

Long-Running Workers and Indexers

Worker and Indexer processes are long-running services — they are not started or stopped by the Runner. A Runner invocation triggers a single batch run and exits when it completes, but the Workers and Indexers that processed its documents keep running and are ready to handle the next run immediately.

Sequential runs share the same processes. You can invoke the Runner repeatedly — nightly, hourly, or on demand — against the same pool of Workers and Indexers without restarting them. Each invocation is an independent run with its own accounting; the processes simply continue consuming from the source topic.

Concurrent runs are supported. Multiple Runner invocations can be active simultaneously, all handled by the same Workers and Indexers. Documents from different runs are interleaved on the Kafka source topic and processed without coordination or interaction between runs.

Multiple pipelines can coexist. A single Lucille config can define multiple connectors feeding different pipelines. In distributed mode, each pipeline has its own set of Kafka topics ({pipeline}_source, {pipeline}_dest). You can run separate fleets of Workers and Indexers for each pipeline — start Workers with com.kmwllc.lucille.core.Worker pipeline-a and separately com.kmwllc.lucille.core.Worker pipeline-b. The Runner publishes each connector’s documents to the correct pipeline’s source topic based on the connector’s pipeline config property.

The constraint is: for each pipeline referenced by a connector in your config, Workers and an Indexer must be running for that pipeline before the Runner starts. If a connector references pipeline: "enrichment", there must be Workers consuming from enrichment_source and an Indexer consuming from enrichment_dest.

Multi-connector runs execute connectors sequentially within a single Runner invocation. Each connector’s documents are published to its pipeline’s source topic, and the Runner waits for all of that connector’s documents to reach a terminal state before starting the next connector. If a connector fails, subsequent connectors are skipped.

How run isolation works. The Publisher stamps a unique run_id on every document it publishes. When a Worker or Indexer sends a lifecycle event (FINISH, FAIL, DROP) for a document, it reads the run_id from that document to determine which Kafka event topic to write to. Each run has its own dedicated event topic named {pipeline}_event_{runId}. The Publisher for each Runner invocation only consumes its own event topic, so completion accounting is completely isolated — concurrent runs do not interfere with each other.

3 - Distributed Streaming

Streaming Mode (Connectorless)

In streaming mode, Lucille runs without a Runner or Connector. An external system places documents onto a Kafka source topic directly, and one or more Worker or WorkerIndexer processes consume and process them continuously — with no run boundary or completion accounting.

When to use: Keeping a search index in sync with a live system (e.g., consuming from a CDC topic, Debezium, or any Kafka producer). The pipeline logic is identical to batch mode; only the execution model changes.

Starting Workers in Streaming Mode

java \
  -Dconfig.file=/path/to/config.conf \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.Worker \
  my-pipeline-name

Or use WorkerIndexer to co-locate processing and indexing in one process:

java \
  -Dconfig.file=/path/to/config.conf \
  -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*' \
  com.kmwllc.lucille.core.WorkerIndexer \
  my-pipeline-name

External documents must be placed onto the topic named {pipeline-name}_source (or the topic named by kafka.sourceTopic if set).

Consuming from Multiple Topics with a Pattern

When running a WorkerIndexer in streaming mode, kafka.sourceTopic is interpreted as a Java regex pattern rather than a literal topic name. This means a single WorkerIndexer process can consume from multiple Kafka topics simultaneously by setting sourceTopic to a pattern that matches all of them:

kafka {
  bootstrapServers: "kafka:9092"
  sourceTopic: "orders_.*_source"   # matches orders_us_source, orders_eu_source, etc.
  events: false
}

Kafka’s consumer group protocol assigns partitions from all matching topics to the consumer. As new matching topics are created, the consumer group rebalances and picks them up automatically.

Note: this behavior applies to WorkerIndexer only. A standalone Worker subscribes to a single exact topic name and does not support pattern matching.

Configuration for Streaming Mode

Streaming mode uses the same pipeline and Kafka configuration as distributed batch mode, with two differences:

• No connectors block — external producers push documents to the Kafka source topic.
• No Publisher — there is no run boundary and no completion accounting.

Run IDs in Streaming Mode

In batch mode, the Publisher stamps a run_id on every document at publication time. In streaming mode there is no Publisher, so documents arriving from an external system typically have no run_id field. This is normal and expected — the run_id is a batch-mode concept.

Consequences of absent run_ids:

• The Worker’s MDC will have no run_id context in log lines (the field is null).
• If events are enabled, the default event topic name would be {pipeline}_event_null (since the topic name is derived from the run_id).

If your external system wants to stamp a run_id on documents (e.g., to correlate a batch of updates), it can include a run_id field in the JSON before placing it on the Kafka topic. The Worker will read it and use it for MDC logging. But this is entirely optional.

Event Configuration for Streaming Mode

Recommendation: Set kafka.events: false unless you have a specific consumer for the event topic. If you do need events, set kafka.eventTopic to a fixed name to avoid the _event_null topic naming issue.

kafka {
  bootstrapServers: "kafka:9092"

  # Option A: Disable events entirely (most common for streaming)
  events: false

  # Option B: Send events to a fixed topic for external monitoring
  # events: true
  # eventTopic: "lucille_events"
}

pipelines: [
  {
    name: "my-pipeline"
    stages: [ ... ]
  }
]

indexer { type: "OpenSearch" }
opensearch { ... }

Batch vs. Streaming: Same Pipeline Logic

The same Stage implementations work in both modes. Develop and test enrichment logic in batch mode (where RunType.TEST and the run summary make correctness verification straightforward), then deploy the same pipeline configuration in streaming mode for real-time production ingestion. For real-world search systems — which typically need an initial batch backfill followed by continuous updates — this means one pipeline definition, not two.

4 - Hybrid Streaming

Hybrid Streaming Mode

In hybrid streaming mode, Lucille runs one or more WorkerIndexer processes that co-locate document processing and indexing in a single JVM. Like distributed streaming, there is no Runner or Connector — an external system places documents onto a Kafka source topic, and the WorkerIndexer processes consume, enrich, and index them continuously.

When to use: Streaming ingestion where you want the horizontal scalability of distributed mode but with simpler operations — fewer processes to manage, no separate Indexer deployment, and reduced network hops between processing and indexing.

5 - Docker Compose

Docker Compose Deployment

Docker Compose is the simplest way to run Lucille in distributed mode. The lucille-distributed-example provides a complete working template. The key structure:

services:
  zookeeper:
    # Bundled with Kafka; required for Kafka's internal coordination
    image: ...

  kafka:
    depends_on: [zookeeper]
    healthcheck:
      test: ["CMD", "kafka-topics.sh", "--bootstrap-server=kafka:9092", "--list"]

  worker:
    depends_on:
      kafka: { condition: service_healthy }
    entrypoint: java -Dconfig.file=/conf/config.conf -cp '/target/lib/*'
                     com.kmwllc.lucille.core.Worker my-pipeline

  indexer:
    depends_on:
      kafka: { condition: service_healthy }
      solr:  { condition: service_healthy }
    healthcheck:
      # Delegate healthcheck to the search backend — Runner waits for this before starting
      test: curl -f 'http://solr:8983/solr/...'
    entrypoint: |
      # Create the collection, then start the Indexer
      curl 'http://solr:8983/solr/admin/collections?action=CREATE&name=quickstart...'
      java -Dconfig.file=/conf/config.conf -cp '/target/lib/*'
           com.kmwllc.lucille.core.Indexer my-pipeline

  runner:
    depends_on:
      kafka:   { condition: service_healthy }
      indexer: { condition: service_healthy }  # Ensures Indexer + backend are ready
    entrypoint: java -Dconfig.file=/conf/config.conf -cp '/target/lib/*'
                     com.kmwllc.lucille.core.Runner -usekafka

Key design points from the example:

• All components share one config file — config.file points to the same .conf for all containers.
• Classpath is target/lib/* — Maven’s dependency:copy-dependencies puts all JARs there; the main lucille JAR and all dependencies are sibling files.
• The Indexer container’s health check proxies the search backend’s health — this causes Docker Compose to hold the Runner until the search backend is ready, without the Runner needing to know anything about it.
• Workers start before the Runner — Kafka consumer group rebalancing takes time; workers should be consuming before the Runner begins publishing documents.

6 - Kubernetes

Kubernetes Deployment

Lucille’s architecture maps naturally onto Kubernetes primitives.

Batch Jobs as Kubernetes CronJobs

For scheduled batch ingests, package Lucille as a container and run it as a CronJob. When the run completes, the container exits — there is no long-running process to manage between runs.

Minimal Dockerfile:

FROM eclipse-temurin:17-jre

WORKDIR /app

# Copy all JARs (lucille-core + dependencies) from the Maven build output
COPY target/lib/ lib/

# Copy your pipeline configuration
COPY conf/ conf/

# The config file path is provided via an environment variable at runtime
ENV CONF=""
ENTRYPOINT ["sh", "-c", "java -Xmx4g -Dconfig.file=${CONF} -cp 'lib/*' com.kmwllc.lucille.core.Runner"]

Build the image after running mvn clean install in your project (which copies all dependencies to target/lib/):

docker build -t my-lucille-image .

Run it:

docker run --env CONF=conf/my-pipeline.conf \
           --env OPENSEARCH_URL=https://opensearch:9200 \
           my-lucille-image

The same image can run any Lucille component by overriding the entrypoint:

# Run as a Worker (distributed mode)
docker run --env CONF=conf/my-pipeline.conf \
           --entrypoint sh my-lucille-image \
           -c "java -Xmx2g -Dconfig.file=\${CONF} -cp 'lib/*' com.kmwllc.lucille.core.Worker my-pipeline"

# Run as an Indexer (distributed mode)
docker run --env CONF=conf/my-pipeline.conf \
           --entrypoint sh my-lucille-image \
           -c "java -Xmx1g -Dconfig.file=\${CONF} -cp 'lib/*' com.kmwllc.lucille.core.Indexer my-pipeline"

CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: lucille-nightly-ingest
spec:
  schedule: "0 2 * * *"  # 2am daily
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: lucille
            image: my-registry/lucille:latest
            env:
            - name: OPENSEARCH_URL
              valueFrom:
                secretKeyRef:
                  name: opensearch-credentials
                  key: url
            resources:
              requests:
                memory: "2Gi"
                cpu: "1"
              limits:
                memory: "4Gi"
                cpu: "4"

Distributed Deployment as Kubernetes Pods

In distributed mode, each Lucille component runs as its own pod:

Worker Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lucille-workers
spec:
  replicas: 4  # Scale by changing replicas
  selector:
    matchLabels:
      app: lucille-worker
  template:
    metadata:
      labels:
        app: lucille-worker
    spec:
      containers:
      - name: worker
        image: my-registry/lucille:latest
        command: ["java", "-cp", "/app/lucille.jar:/app/lib/*",
                  "com.kmwllc.lucille.core.Worker", "my-pipeline"]
        env:
        - name: config.file
          value: /app/config.conf
        resources:
          requests:
            memory: "2Gi"
            cpu: "2"

Workers are the natural scaling target. When the Kafka source topic backlog grows, increase replicas. Kubernetes’ Horizontal Pod Autoscaler can drive this automatically using Kafka consumer group lag as a metric (via KEDA or similar).

7 - Production Operations

Memory Sizing

Rule of thumb: worker.threads × (Stage resource usage). A pipeline that loads a 500MB NLP model uses 500MB × number of Worker threads.

Always set -Xmx explicitly. Avoid letting Java use all available RAM.

Backpressure and Queue Sizing

In local mode, set publisher.queueCapacity to bound the number of in-flight documents:

publisher {
  queueCapacity: 10000
}

In distributed mode, use publisher.maxPendingDocs to throttle the Connector:

publisher {
  maxPendingDocs: 80000
}

Without backpressure, a fast Connector can publish faster than Workers process, causing out-of-memory conditions.

Indexer Batch Tuning

The Indexer sends documents in batches. Both batch size and timeout are independently configurable:

indexer {
  batchSize: 200        # Flush when this many docs accumulate
  batchTimeout: 5000    # Flush after 5 seconds if batchSize is not reached
}

Larger batches improve indexing throughput (fewer API calls). The timeout prevents documents from waiting indefinitely when volume is low.

Graceful Shutdown

Lucille handles SIGINT (Ctrl+C) and SIGTERM gracefully. On receiving a signal:

1. The Connector stops publishing new documents.
2. Workers drain the remaining documents.
3. The Indexer flushes its current batch.
4. The process exits with a run summary.

Do not send SIGKILL — documents in flight may not be indexed.

Monitoring

See Logging for setting up log monitoring.

Key metrics logged periodically during a run:

INFO PublisherImpl: 37029 docs published. One minute rate: 3225.69 docs/sec. Waiting on 21014 docs.
INFO WorkerPool: 27017 docs processed. One minute rate: 1787.10 docs/sec. Mean pipeline latency: 10.63 ms/doc.
INFO Indexer: 17016 docs indexed. One minute rate: 455.07 docs/sec. Mean backend latency: 6.90 ms/doc.

The frequency is controlled by log.seconds (default: 30).

Run Summary

At the end of every run, Lucille prints a structured summary:

RUN SUMMARY: Success. 1/1 connectors complete. All published docs succeeded.
connector1: complete. 200000 docs succeeded. 0 docs failed. 0 docs dropped. Time: 416.47 secs.
Run took 417.46 secs.

A connector that failed entirely is distinguished from one that completed with individual document failures. Subsequent connectors after a failure are listed as skipped.

Production Checklist

• Set -Xmx heap limit appropriate for your pipeline’s memory usage.
• Configure publisher.queueCapacity (local mode) or publisher.maxPendingDocs (distributed).
• Tune indexer.batchSize and indexer.batchTimeout for your backend’s throughput.
• Use environment variable substitution for credentials (${?VAR_NAME}) — never hard-code secrets in config files.
• Route DocLogger output to a separate file in production (see Logging).
• Enable runner.metricsLoggingLevel: "INFO" for stage-by-stage metrics at run completion.
• Configure worker.maxRetries and ZooKeeper if poison-pill protection is needed — without this, a document that repeatedly crashes a Worker will stall the pipeline indefinitely. Failed documents are routed to the {pipeline}_fail topic for inspection and replay.
• Set runner.connectorTimeout if any connector might run longer than 24 hours (the default timeout).