Operations Guide on Lucille

Configuration Management

Mon, 09 Jun 2025 00:00:00 +0000

This page covers the operational patterns for working with Lucille’s configuration system — how to use environment variable substitution, how to compose configs from reusable files, how to deploy configs in containers, and how to validate configs before running. For the architectural rationale behind Lucille’s choice of HOCON and the Typesafe Config library, see Architecture: Config.

Environment Variable Substitution

In containerized deployments (Docker, Kubernetes), credentials and environment-specific settings are typically provided via container environment variables or mounted secrets. A database password, an API key, a search engine URL — these should not be hardcoded in a config file that lives in version control.

Logging Setup

Wed, 07 May 2025 00:00:00 +0000

Lucille Logs

Lucille uses Log4j2 (via SLF4J) for all logging. A default log4j2.xml is included in lucille-core/src/main/resources/.

Using a Custom Log4j2 Configuration

To override the default logging configuration, pass the standard Log4j2 system property on the Java command line:

java -Dlog4j.configurationFile=/path/to/my-log4j2.xml \
 -Dconfig.file=my-pipeline.conf \
 -cp 'target/lib/*' \
 com.kmwllc.lucille.core.Runner

This applies to all Lucille components (Runner, Worker, Indexer, WorkerIndexer). If you don’t provide this property, Log4j2 uses whatever log4j2.xml it finds on the classpath.

Log Interpretation

Mon, 09 Jun 2025 00:00:00 +0000

Log Format

Lucille uses SLF4J with Log4j2 as the logging backend. The log format depends on your Log4j2 configuration:

Plain text (typical console/file output):

2026-05-06 20:51:53.581 INFO [main] c.k.l.c.Runner - Pipeline Configuration is valid.

JSON (structured logging for log aggregation):

{
 "@timestamp": "2026-05-06T20:51:53.581Z",
 "log.level": "INFO",
 "message": "Pipeline Configuration is valid.",
 "process.thread.name": "main",
 "log.logger": "com.kmwllc.lucille.core.Runner",
 "run_id": "c1d9413a-..."
}

The JSON format includes the run_id and id (document ID) fields from the MDC, making it possible to filter logs by run or by document in log aggregation tools. In plain text mode, only the message field is typically visible.

Performance Tuning

Mon, 09 Jun 2025 00:00:00 +0000

Before You Optimize: Choosing the Right Deployment Mode

Distributed mode provides a pathway to higher throughput by running multiple Worker and Indexer processes across machines. However, there is inherent overhead in serializing and deserializing documents across Kafka and process boundaries — every document is converted to JSON, written to Kafka, read from Kafka, and parsed back into a Document object at each hop.

Distributed mode is not necessary or justified for all projects. For a small ingestion task (thousands to low millions of documents with lightweight enrichment), local single-JVM mode is often sufficient and faster end-to-end than distributed mode because it avoids all serialization overhead. The in-memory queues in local mode have near-zero latency compared to Kafka round-trips.

Security Configuration

Mon, 09 Jun 2025 00:00:00 +0000

Lucille connects to external systems — search backends (Solr, OpenSearch, Elasticsearch, Pinecone) and Kafka — that may require TLS encryption and authentication. This page covers how to configure security for each.

General Principles

Never hard-code credentials in config files. Use HOCON’s environment variable substitution to inject secrets at runtime:

opensearch {
 url: "https://localhost:9200"
 url: ${?OPENSEARCH_URL} # override from env var
}

In Kubernetes, inject credentials via Secrets mounted as environment variables. In Docker, use --env or --env-file. The config file can live in version control with defaults; secrets come from the environment.

REST API

Mon, 09 Jun 2025 00:00:00 +0000

The lucille-api plugin adds an HTTP REST API built on Dropwizard that allows managing configs and triggering runs over HTTP rather than via the CLI. It includes a Swagger UI and optional basic authentication.

Current limitation: The REST API launches all runs in local mode (in-memory queues, Workers and Indexer as threads within the API server’s JVM). It cannot currently trigger a distributed Kafka-based ingest. If you need distributed mode, use the CLI Runner with -usekafka.

Support Matrix

Mon, 09 Jun 2025 00:00:00 +0000

Java

Version	Status
Java 21	Supported (minimum required)

Lucille is compiled targeting Java 21. This version is tested in CI via GitHub Actions using Eclipse Temurin distributions.

Build Tool

Apache Maven 3.x is required to build Lucille from source.

Search Backends

The following backends are supported via the Indexer component.

Backend	Tested Version	Notes
Apache Solr	9.x	Client: `solr-solrj` 9.8.0
OpenSearch	2.x	Client: `opensearch-java` 2.11.1
Elasticsearch	8.x	Client: `elasticsearch-java` 8.18.4
CSV	—	Writes indexed documents to a local CSV file; no server required
Nop	—	Discards output; used for testing and dry runs

Kafka (Distributed Mode)

Component	Version
kafka-clients	4.0.0
Kafka broker	3.x, 4.x

Kafka is only required when running in distributed mode (-usekafka). Local mode uses in-memory queues and has no Kafka dependency.

Troubleshooting

Tue, 15 Oct 2024 00:00:00 +0000

If something isn’t working, read logs first. Lucille logs are detailed and will usually identify where the problem is coming from. See Log Analysis for a guide to reading Lucille logs.

Debugging Failed Documents

The run summary reports how many documents failed, but not which ones or why. Here’s the workflow for investigating failures.

Step 1: Find the failed document IDs and reasons

Search the logs for failure messages. These are logged at ERROR level by the DocLogger. All document failures use a standardized "Document FAILED" prefix, so you can find every failure in one pass and then drill into specific types: