Component Developer Guide on Lucille

What the Framework Gives You

Mon, 09 Jun 2025 00:00:00 +0000

When you are developing components for Lucille, it is important to understand what the framework does and does not give you. That understanding lets you take full advantage of what is already built, and gives you realistic expectations about the responsibilities that will fall to you as the implementor.

This page begins with a general discussion of what you get when you adopt the framework overall. The following sections then focus on each component type — Stage, Connector, and Indexer. When you implement a component you will be extending the Stage or Indexer abstract classes, or implementing the Connector interface. Each section explains how the framework interacts with that component, what the base class or framework infrastructure does for you automatically, and what you are responsible for writing yourself.

Developing New Components

Mon, 28 Oct 2024 00:00:00 +0000

Each component type has its own dedicated guide:

Developing Stages — Skeleton, lifecycle, conditional execution, and the Document API.
Developing Connectors — Skeleton, lifecycle, and publishing documents.
Developing Indexers — Skeleton, lifecycle, and sending documents to a destination.

All components must declare a SPEC and follow the Javadoc Standards. See Testing Pipelines for testing conventions.

Developing Stages

Mon, 28 Oct 2024 00:00:00 +0000

To create a Stage, extend the abstract Stage class and implement processDocument(). That is the only method you are required to provide. The base class handles everything else: config validation, condition evaluation, metrics, thread isolation, and error routing.

What the base class does for you:

Config validation — The constructor calls getSpec().validate(config) using your class’s SPEC field. If the config has missing required properties, unknown properties, or type mismatches, validation fails at startup with a clear error message. You never call this yourself.
Condition evaluation — If the user configures conditions on your stage, the base class evaluates them before calling processDocument(). If conditions are not met, your method is never invoked for that document. You write processDocument() as if conditions are always satisfied.
Thread isolation — Each worker thread gets its own instance of your stage. Instance fields are effectively thread-local; no synchronization is needed.
Metrics — The base class tracks per-stage document count, latency, error count, and child count automatically.
Error handling — If processDocument() throws a StageException, the framework catches it, marks the document as failed, and continues processing other documents.

What you implement:

Developing Connectors

Mon, 28 Oct 2024 00:00:00 +0000

To create a Connector, extend AbstractConnector and implement execute(). That is the only method you are required to provide. The execute() method receives a Publisher — your connector reads from its data source and calls publisher.publish(doc) for each document it produces. That’s the core job of a connector: create documents and publish them. The base class handles config validation, name/pipeline resolution, document ID prefixing, and provides no-op defaults for the optional lifecycle methods.

Developing Indexers

Mon, 28 Oct 2024 00:00:00 +0000

To create an Indexer, extend the abstract Indexer class and implement three methods: sendToIndex(), validateConnection(), and closeConnection(). The base class manages the entire consumption loop — polling documents from the indexing queue, accumulating them into batches, flushing batches on size or timeout, sending completion/failure events, retrying transient failures, and logging throughput metrics.

Indexer is an abstract class (like Stage) rather than an interface because the framework drives the indexer’s execution. The base class implements Runnable and its run() method contains the message consumption loop, batching logic, retry machinery, and event accounting. Your implementation only provides the transport-specific operations: validate the connection, send a batch, and close the connection.

Developing File Handlers

Fri, 23 May 2025 00:00:00 +0000

A FileHandler turns a file’s content into Documents. Each implementation handles a specific file format — CSV, JSON, XML, or any custom format you need. The FileConnector uses FileHandlers to parse files it discovers during traversal.

To create a FileHandler, implement the FileHandler interface. The framework handles discovery, instantiation, and integration with the FileConnector.

What the framework does for you:

Discovery by file extension — FileHandlers are mapped to file extensions in the connector’s fileHandlers config block. The framework instantiates your handler and routes files to it based on their extension.
Config validation — Your getSpec() method declares legal properties; the framework validates the config at startup.
InputStream management — The framework opens the file (from local disk, S3, Azure, GCS, or inside archives) and passes you an InputStream. You don’t need to know where the file came from.

What you implement:

Developing Storage Clients

Fri, 23 May 2025 00:00:00 +0000

A StorageClient traverses a storage system, discovers files, and provides access to their content. The FileConnector uses StorageClients to list and read files from local disk, S3, Azure Blob Storage, and Google Cloud Storage. If you need to read files from a storage system that Lucille doesn’t support out of the box (e.g., SFTP, SMB/CIFS, WebDAV), you implement a StorageClient.

Note: It is not currently possible to reference a custom StorageClient from a Lucille config. The mapping from URI scheme to StorageClient implementation is hardcoded in StorageClient.create(). Adding a new storage backend today requires a PR to lucille-core — see the Contributor Guide for project structure and contribution workflow. Config-driven pluggable StorageClients are a roadmap feature. This page is included in the Component Developer Guide because the implementation pattern is the same regardless of how the client is registered.

SPEC Validation System

Mon, 09 Jun 2025 00:00:00 +0000

Overview

The SPEC system is Lucille’s configuration validation framework. It catches config errors — typos, missing required fields, wrong types — before any processing starts. Every Stage, Connector, and Indexer declares a public static final Spec SPEC that defines what configuration properties it accepts.

For example, this is how a Stage would declare that its config must contain a string property called foo:

public static final Spec SPEC = SpecBuilder.stage().requiredString("foo").build();

The philosophy: fail loudly at startup, not silently at runtime. If a user’s config omits a required property or includes an unrecognized one (like a typo), Lucille reports the error before any documents are processed.

Quick Reference

Mon, 09 Jun 2025 00:00:00 +0000

This page provides copy-paste-ready examples for the patterns developers use most frequently. It complements the detailed explanations in the Architecture and Contributing sections.

The Threading Model

Each Worker thread creates its own Pipeline object, which creates its own instance of every Stage. Instance fields in a Stage are effectively thread-local — use them freely without synchronization.

When to use a singleton: If a resource is both expensive to initialize and thread-safe for concurrent use, share it via a static field with lazy initialization. The existing DictionaryManager class is the canonical pattern.

Testing Pipelines

Mon, 09 Jun 2025 00:00:00 +0000

Lucille provides a first-class test mode that lets you run a complete pipeline end-to-end against real source data, without needing a running search backend. All documents, events, and messages are captured in memory and available for assertion after the run.

RunType.TEST

When a run is started with RunType.TEST, Lucille:

Runs all components (Connector, Workers, Indexer) as normal.
Bypasses the search backend — no actual indexing occurs.
Captures all messages flowing between components in a TestMessenger.
Returns a RunResult containing the captured message history for assertions.

Running in Test Mode

Use Runner.runInTestMode(config) or construct a Runner with RunType.TEST:

Javadoc

Mon, 01 Jan 0001 00:00:00 +0000

This page covers two things: where to find the published Javadoc for the Lucille API, and how to write Javadoc on new Connectors, Stages, and Indexers so that the documentation tooling can parse and render their config parameters correctly.

Published API Reference

The generated Javadoc for lucille-core is published at:

javadoc.io/doc/com.kmwllc/lucille-core

This covers all public classes, interfaces, and methods in the core library, including the Document, Stage, Connector, Indexer, and Publisher APIs.