This is the multi-page printable view of this section. Click here to print.
This section is for developers contributing code back to the Lucille project. It covers how the project is structured, how to set up your development environment, and the coding conventions used throughout the codebase.
If you are writing components for your own project rather than contributing to Lucille itself, see the Component Developer Guide instead.
Lucille is a multi-module Maven project. The module hierarchy is:
lucille (root aggregator)
├── lucille-parent # parent POM: dependency versions, plugin config, build profiles
├── lucille-bom # Bill of Materials: version-aligned dependency declarations
├── lucille-core # the framework itself: Runner, Worker, Indexer, Pipeline, Document, Stages, Connectors
├── lucille-plugins/ # optional modules with heavy or specialized dependencies (tika, pinecone, weaviate, jlama, parquet, ocr, video, entity-extraction, api)
└── lucille-examples/ # runnable example projects demonstrating common ingestion patterns (not published to Maven Central)
The parent POM that all other modules inherit from. It defines:
deploy profile for publishing to Maven Central with GPG signing)When should you be concerned with lucille-parent? When you need to:
dependencyManagement)Most day-to-day development (writing stages, connectors, tests) does not require touching lucille-parent.
The Bill of Materials POM. It declares all Lucille modules (lucille-core, lucille-pinecone, lucille-tika, etc.) with their versions aligned to ${project.version}. External projects that depend on Lucille import the BOM in their dependencyManagement section, which lets them declare Lucille dependencies without specifying versions:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.kmwllc</groupId>
<artifactId>lucille-bom</artifactId>
<version>0.9.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.kmwllc</groupId>
<artifactId>lucille-core</artifactId>
<!-- version inherited from BOM -->
</dependency>
<dependency>
<groupId>com.kmwllc</groupId>
<artifactId>lucille-tika</artifactId>
<!-- version inherited from BOM -->
</dependency>
</dependencies>
The BOM ensures that all Lucille modules in a project are at the same version, preventing subtle incompatibilities.
The framework itself. Contains:
Runner, Worker, Indexer, Publisher, Pipeline, DocumentLocalMessenger, TestMessenger, KafkaWorkerMessenger, etc.TestMessenger, RunType.TEST)This is the only module required for a minimal Lucille deployment. If your pipeline doesn’t need Tika, Pinecone, OCR, etc., you only need lucille-core.
An aggregator POM containing optional modules. Each plugin:
pom.xml with lucille-core as a provided dependencyThe provided scope on lucille-core means the plugin compiles against core but does not bundle it — at runtime, core is expected to already be on the classpath. This prevents version conflicts when multiple plugins are used together.
When to create a new plugin module:
Runnable example projects that demonstrate common ingestion patterns. Each example:
pom.xml importing the lucille-bomlucille-core and whichever plugins it needsconf/ directory with HOCON config filesscripts/ directory with shell scripts for runningmaven-dependency-plugin to copy runtime dependencies to target/lib/Examples are not published to Maven Central (<maven.deploy.skip>true</maven.deploy.skip>). They exist purely as reference implementations and starting points for new projects.
The project version is specified in lucille-parent/pom.xml:
<version>0.9.0-SNAPSHOT</version>
All other modules inherit this version via their <parent> declaration. The version appears explicitly in each module’s parent reference (Maven requires this), but the single source of truth is lucille-parent.
SNAPSHOT vs. release: During development, the version ends with -SNAPSHOT. When a release is cut, the version is updated to remove -SNAPSHOT (e.g., 0.9.0), the release is built and published, and then the version is bumped to the next SNAPSHOT (e.g., 0.10.0-SNAPSHOT).
Updating the version: Because Maven requires the version to be explicit in each module’s <parent> block, updating the version requires changing it in every pom.xml in the project. This is typically done with the Maven versions plugin:
mvn versions:set -DnewVersion=0.9.0
From the root directory:
mvn clean install
This builds all modules in dependency order: lucille-parent → lucille-bom → lucille-core → lucille-plugins (each plugin) → lucille-examples (each example). Each module produces a JAR artifact installed to the local Maven repository.
mvn clean install -pl lucille-core
Or for a plugin:
mvn clean install -pl lucille-plugins/lucille-tika
target/lucille.jar — a thin JAR containing only Lucille’s own classes (named lucille.jar via finalName in the POM). Runtime dependencies are copied to target/lib/. To run Lucille, both must be on the classpath: -cp 'target/lucille.jar:target/lib/*' (or simply -cp 'target/lib/*' since the lucille JAR is also copied there).target/lib/ containing all runtime dependencies (via maven-dependency-plugin copy-dependencies).| Module | JAR Type | Why |
|---|---|---|
lucille-core | Thin JAR + target/lib/ | Core is always on the classpath alongside its dependencies. No need to shade. |
| Plugins (tika, pinecone, etc.) | Shaded (fat) JAR | Plugins bundle their heavyweight dependencies to avoid version conflicts with core or other plugins. A single plugin JAR can be dropped onto the classpath without worrying about transitive dependency management. |
| Examples | Thin JAR + target/lib/ | Examples are runnable projects, not libraries. Dependencies are copied for easy classpath setup. |
When running Lucille from the command line, the classpath typically looks like:
java -Dconfig.file=config.conf -cp 'lucille-core/target/lucille.jar:lucille-core/target/lib/*:lucille-plugins/lucille-tika/target/lucille-tika-*.jar' com.kmwllc.lucille.core.Runner
Or more simply from an example directory where everything is in target/lib/:
java -Dconfig.file=conf/my-config.conf -cp 'target/lib/*' com.kmwllc.lucille.core.Runner
After building, from an example’s directory:
java -Dconfig.file=conf/file-to-file-example.conf -cp 'target/lib/*' com.kmwllc.lucille.core.Runner
If you write your own Connector or Stage in a separate Maven project, you need to:
<dependencyManagement>
<dependencies>
<dependency>
<groupId>com.kmwllc</groupId>
<artifactId>lucille-bom</artifactId>
<version>0.9.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>com.kmwllc</groupId>
<artifactId>lucille-core</artifactId>
</dependency>
</dependencies>
Implement your Stage or Connector following the standard patterns (SPEC declaration, constructor calling super(config), etc.)
Build your project to produce a JAR.
Include your JAR on the classpath when running Lucille. The simplest approach is to copy your JAR into the target/lib/ directory alongside Lucille’s JARs.
Reference your class in the config using its fully qualified class name:
stages: [
{
class: "com.mycompany.lucille.stages.MyCustomStage"
myParam: "value"
}
]
Lucille instantiates Stages, Connectors, and Indexers reflectively using the class property in the config. As long as the class is on the classpath and follows the expected constructor signature, it will work. There is no registration step, no plugin manifest, no service loader — just put the class on the classpath and reference it by name.
Lucille’s “plugin system” is simply the JVM classpath. To add a capability:
This is why the built-in plugins are structured as separate JARs rather than using a framework-specific plugin API. A plugin JAR is just a JAR with classes that happen to extend Lucille’s base classes. Your custom code works exactly the same way.
Each module follows standard Maven conventions:
lucille-core/
├── src/
│ ├── main/
│ │ ├── java/com/kmwllc/lucille/
│ │ │ ├── core/ # Runner, Worker, Indexer, Publisher, Pipeline, Document, Stage
│ │ │ ├── connector/ # built-in Connectors
│ │ │ ├── indexer/ # IndexerFactory and related utilities
│ │ │ ├── message/ # Messenger interfaces and implementations
│ │ │ ├── stage/ # built-in Stages
│ │ │ └── util/ # utilities (ConfigUtils, LogUtils, etc.)
│ │ └── resources/
│ │ ├── reference.conf # default config values
│ │ └── validConfigProperties.conf # config validation rules
│ └── test/
│ ├── java/com/kmwllc/lucille/ # test classes
│ └── resources/ # test config files
└── pom.xml
lucille-core/src/main/java/com/kmwllc/lucille/stage/lucille-core/src/main/java/com/kmwllc/lucille/connector/lucille-plugins/lucille-examples/src/test/resources/ in the relevant modulelucille-core produces a test JAR (lucille-core-0.9.0-SNAPSHOT-tests.jar) that is available to other modules for testing. The examples module depends on this test JAR, which provides test utilities and base classes for writing integration tests against the framework.
Lucille is published to Maven Central via Sonatype OSSRH. The deploy profile in lucille-parent activates GPG signing and source/Javadoc JAR generation:
mvn clean deploy -Ddeploy
This publishes lucille-core, lucille-bom, and all plugin modules. Examples are excluded from publishing (maven.deploy.skip=true).
| Module | Purpose | Published? |
|---|---|---|
| lucille-parent | Version management, dependency versions, build config | Yes |
| lucille-bom | Version-aligned dependency declarations for consumers | Yes |
| lucille-core | The framework (all core components) | Yes |
| lucille-plugins/* | Optional modules with heavy dependencies | Yes (each) |
| lucille-examples/* | Runnable reference implementations | No |
The key things a developer needs to know:
provided scope for core. They compile against it but don’t bundle it.Assuming that we don’t want to auto-format non-java files via a directory level ‘Reformat Code’ option, we need to exclude all other files from being reformatted
Navigate to Settings | Preferences in Intellij IDEA
Navigate to Editor → Code Style
Click on the tab on the right window labeled ‘Formatter’
In the ‘Do Not Format’ text box, paste the following and click ‘Apply'
*.{yml,xml,md,json,yaml,jsonl,sql}
A restart of Intellij may be required to see changes
This method may prove to be too complicated, especially when new file types are added to the codebase, therefore, consider the following, simpler method instead:
Eclipse import conf .xml files
The linked post details some useful information for how Eclipse users can use the same .xml for their code formatting on Eclipse IDE.
Lucille follows conventions broadly consistent with the Google Java Style Guide, with some deviations noted below. There is no enforced checkstyle or formatter plugin in the build — conventions are maintained through code review.
Indentation: 2 spaces. All source files use 2-space indentation, consistent with Google style. No tabs.
Line length: No strict enforced limit, but lines are generally kept under 120 characters. Long lines (especially in constructors with many parameters or complex conditionals) are broken at natural points.
Braces: Opening braces on the same line as the statement (K&R style). Closing braces on their own line. Single-statement if blocks still use braces in most cases, though there are occasional instances of brace-less single-line if statements (a minor deviation from strict Google style).
if (doc == null) {
commitOffsetsAndRemoveCounter(null);
continue;
}
Blank lines: One blank line between methods. One blank line between logical sections within a method. No multiple consecutive blank lines (though occasional double blanks appear in older code).
Classes: PascalCase. Stage names describe what they do: CopyFields, DeleteFields, RenameFields, ChunkText, EmitNestedChildren. Connectors are named for their source: FileConnector, DatabaseConnector, SolrConnector. Indexers are named for their destination: SolrIndexer, OpenSearchIndexer, PineconeIndexer.
Methods: camelCase. Standard Java conventions. Getters use get prefix (getId(), getString()). Boolean getters use is or has prefix (isDropped(), hasChildren(), has()).
Constants: UPPER_SNAKE_CASE for static final fields that are true constants:
public static final String ID_FIELD = "id";
public static final String RUNID_FIELD = "run_id";
public static final int DEFAULT_BATCH_SIZE = 100;
Instance fields: camelCase with private final where possible. Fields are declared at the top of the class, after constants and the logger:
public class CopyFields extends Stage {
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
public static final Spec SPEC = SpecBuilder.stage()...;
private final Map<String, Object> fieldMapping;
private final UpdateMode updateMode;
private final boolean isNested;
Local variables: camelCase. Short, descriptive names. Loop variables use standard conventions (i, doc, node, entry).
Packages: All under com.kmwllc.lucille. Sub-packages by function: core, stage, connector, indexer, message, util.
The standard ordering within a class is:
static final constants (UPPER_SNAKE_CASE)public static final Spec SPEC declarationstart() method (for Stages)processDocument() or execute() (the main logic)stop() or close() (cleanup)This ordering is consistent across Stages, Connectors, and Indexers.
Two logger patterns are used:
Standard class logger — for operational messages:
private static final Logger log = LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
or equivalently:
private static final Logger log = LoggerFactory.getLogger(FileConnector.class);
Both patterns appear in the codebase. The MethodHandles.lookup().lookupClass() pattern avoids hardcoding the class name (useful when copy-pasting). The explicit class reference is more common in older code.
DocLogger — for per-document lifecycle events (used in core components, not in stages):
private static final Logger docLogger = LoggerFactory.getLogger("com.kmwllc.lucille.core.DocLogger");
All public Stages, Connectors, and Indexers have class-level Javadoc describing what the component does and listing its config parameters in a structured format:
/**
* Copies values from a source field to a destination field based on the field mapping.
*
* <p>
* Config Parameters -
* <ul>
* <li>fieldMapping (Map<String, Object>) : A mapping of source field names to destination field names.</li>
* <li>updateMode (String, Optional) : Determines how writing will be handled. Defaults to 'overwrite'.</li>
* <li>isNested (Boolean, Optional) : Sets whether to treat field names as nested json paths. Defaults to false.</li>
* </ul>
*/
This is a project convention: every Stage documents its config parameters in Javadoc using the Config Parameters - heading with a <ul> list. The format includes the parameter name, its type, whether it’s optional, and a description.
Interface methods have Javadoc. The Document, Publisher, Connector, and Messenger interfaces are thoroughly documented with @param, @return, and behavioral descriptions.
Private methods and implementation details generally do not have Javadoc, though complex private methods sometimes have inline comments explaining the approach.
Test framework: JUnit 4. The project uses JUnit 4 (org.junit.Test, @Before, @After, etc.), not JUnit 5. This is a notable choice — most new Java projects use JUnit 5, but Lucille has remained on JUnit 4.
Test class naming: {ClassName}Test. Tests for CopyFields are in CopyFieldsTest. Tests for SolrIndexer are in SolrIndexerTest.
Test method naming: test{Behavior} in camelCase: testCopyFieldsReplace(), testDeleteFields(), testGetLegalProperties(). This follows the older JUnit 4 convention rather than the more descriptive should_behavior_when_condition style.
Test config files: Stored in src/test/resources/ in a subdirectory matching the test class name: src/test/resources/CopyFieldsTest/replace.conf. Loaded via StageFactory:
private StageFactory factory = StageFactory.of(CopyFields.class);
Stage stage = factory.get("CopyFieldsTest/config.conf");
Assertions: Use JUnit 4 static imports: assertEquals, assertFalse, assertNull, assertThrows. Assertion messages are used for complex assertions but often omitted for simple ones.
Mocking: Mockito is used for mocking external dependencies (HTTP clients, Kafka consumers, Solr clients). The core Lucille components are generally not mocked — test mode provides the full system running in-memory.
No enforced formatter. There is no checkstyle plugin or auto-formatter in the build. Formatting is maintained by convention and code review. This means minor inconsistencies exist (e.g., occasional extra blank lines, slightly varying import ordering).
Import ordering is not strictly enforced. Google style prescribes a specific import order (static imports first, then by package). Lucille generally groups imports logically but does not enforce a strict order. IDE auto-import ordering varies between contributors.
Wildcard imports appear occasionally. Google style discourages wildcard imports (import java.util.*). Lucille generally uses explicit imports but wildcards appear in some files (e.g., import java.util.* in Runner.java).
Line length is relaxed. Google style enforces 100 characters. Lucille allows longer lines, particularly for log messages, exception messages, and method signatures with many parameters.
JUnit 4 rather than JUnit 5. Not a style deviation per se, but notable for developers expecting modern JUnit conventions.
private static final logger named log rather than logger. Google style doesn’t prescribe logger naming, but many Java projects use logger. Lucille consistently uses log.
Field declarations sometimes lack explicit access modifiers in interfaces. Interface fields in Document are implicitly public static final without writing it out (e.g., String ID_FIELD = "id"). This is valid Java but some style guides prefer explicit modifiers.
Config reading pattern: Required parameters use direct Typesafe Config getters. Optional parameters use ConfigUtils.getOrDefault:
this.sourceField = config.getString("sourceField"); // required
this.destField = ConfigUtils.getOrDefault(config, "destField", "output"); // optional with default
Exception handling in stages: Stages throw StageException for errors that should fail the current document. They do not catch and swallow exceptions silently. The Worker handles the exception and routes the document to a failure state.
Immutable fields where possible: Constructor-assigned fields are declared final. Mutable state is used only when necessary (e.g., counters, caches initialized in start()).
No Lombok. The project does not use Lombok or other annotation processors for boilerplate reduction. All getters, constructors, and builders are written explicitly.
Java 17 features: The project targets Java 17 and uses features like text blocks, var (sparingly), List.of(), Map.of(), and pattern matching in instanceof where appropriate.