Local Batch
Running all Lucille components in a single JVM with in-memory queues.
Deployment
How to run Lucille in batch, streaming, and hybrid environments.
Choose a deployment mode based on your scale requirements. The same pipeline configuration runs in all modes — switching modes requires only a command-line flag change, not a code change.
| Mode | When to Use | Command |
|---|---|---|
| Local Batch | Development, small jobs (< millions of docs) | java -cp ... com.kmwllc.lucille.core.Runner |
| Distributed Batch | Production scale-out with multiple workers | Separate Runner, Worker, and Indexer processes |
| Distributed Streaming | Continuous ingestion without a Runner | Separate Worker and Indexer processes |
| Hybrid Streaming | Streaming with co-located processing and indexing | WorkerIndexer processes |
| Deployment Pattern | Details |
|---|---|
| Docker Compose | Quick distributed setup with all components in containers |
| Kubernetes | Production at scale with CronJobs, Deployments, and HPA |
| Production Operations | Memory sizing, backpressure, graceful shutdown, monitoring |
Distributed Batch
Running Lucille components as separate processes communicating via Kafka.
Distributed Streaming
Running Lucille without a Runner for continuous ingestion from Kafka.
Hybrid Streaming
Running WorkerIndexer processes for streaming ingestion with co-located processing and indexing.
Docker Compose
Running Lucille in distributed mode using Docker Compose.
Kubernetes
Deploying Lucille on Kubernetes as CronJobs and scalable pod deployments.
Production Operations
Memory sizing, backpressure, batch tuning, graceful shutdown, monitoring, and the production checklist.