Stages on Lucille

All Stages

Mon, 09 Jun 2025 00:00:00 +0000

This page lists all Stages available in lucille-core and as optional plugin modules.

All stages share common configuration parameters:

Parameter	Description
`class`	Required. The fully qualified class name of the Stage.
`name`	Optional display name used in logging and metrics.
`conditions`	Optional list of conditions controlling when the Stage executes.
`conditionPolicy`	`"any"` or `"all"` (default: `"all"`).

Field Manipulation

CopyFields

com.kmwllc.lucille.stage.CopyFields

Copies one or more source fields to destination fields.

Parameter	Type	Required	Description
`source`	List<String>	Yes	Source field names.
`dest`	List<String>	Yes	Destination field names (parallel to `source`).
`updateMode`	String	No	`overwrite`, `append`, or `skip`. Default: `overwrite`.

{ class: "com.kmwllc.lucille.stage.CopyFields", source: ["title"], dest: ["title_copy"] }

RenameFields

com.kmwllc.lucille.stage.RenameFields

ChunkText

Mon, 09 Jun 2025 00:00:00 +0000

The ChunkText Stage splits a long text field into smaller, optionally overlapping segments. Each segment is emitted as a child document that flows independently through all downstream stages and is indexed as its own record. This is the foundation of retrieval-augmented generation (RAG) pipelines in Lucille.

Configuration

{
 class: "com.kmwllc.lucille.stage.ChunkText"
 source: "body"
 dest: "text"
 chunkingMethod: "sentence"
 chunksToMerge: 5
 chunksToOverlap: 1
 cleanChunks: true
}

Configuration Parameters

Parameter	Type	Required	Description
`source`	String	Yes	Field containing the text to chunk.
`dest`	String	No	Field name for chunk content in child docs. Default: `text`.
`chunkingMethod`	String	No	Chunking strategy. Default: `sentence`. See below.
`regex`	String	Required for `custom`	Regex pattern to split on.
`lengthToSplit`	Integer	Required for `fixed`	Number of characters per chunk.
`chunksToMerge`	Integer	No	How many initial chunks to merge into one final chunk. Default: `1` (no merging).
`chunksToOverlap`	Integer	No	Number of chunks from the previous final chunk to prepend to the current one.
`overlapPercentage`	Integer	No	Percentage of the current chunk’s characters to add from its neighbours. Default: `0`.
`characterLimit`	Integer	No	Hard maximum character count for a final chunk after all merging.
`preMergeMinChunkLen`	Integer	No	Initial chunks shorter than this are appended to a neighbour before merging.
`preMergeMaxChunkLen`	Integer	No	Initial chunks longer than this are truncated before merging.
`cleanChunks`	Boolean	No	Remove internal newlines and trim whitespace from each chunk. Default: `false`.

Chunking Methods

`sentence` (default)

Detects sentence boundaries using an Apache OpenNLP sentence detector. This produces semantically coherent chunks. Best used with chunksToMerge to combine a few sentences into each final chunk.

PromptOllama

Fri, 06 Jun 2025 00:00:00 +0000

What if you could just, actually, put an LLM on everything?

Ollama

Ollama allows you to run a variety of Large Language Models (LLMs) with minimal setup. You can also create custom models using Modelfiles and system prompts.

The PromptOllama Stage allows you to connect to a running instance of Ollama Server, which communicates with an LLM through a simple API. The Stage sends part (or all) of a Document to the LLM for generic enrichment. You’ll want to create a custom model (with a Modelfile) or provide a System Prompt in the Stage Config that is tailored to your pipeline.

QueryOpensearch

Fri, 06 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.stage.QueryOpensearch

OpenSearch Templates

You can use templates in OpenSearch to repeatedly run a certain query using different parameters. For example, if we have an index full of parks, and we want to search for a certain park, we might use a template like this:

{
 "source": {
 "query": {
 "match_phrase": {
 "park_name": "{{park_to_search}}"
 }
 }
 }
}

In Opensearch, you could then call this template (providing it park_to_search) instead of writing out the full query each time you want to search.

EmbeddedPython

Tue, 16 Dec 2025 00:00:00 +0000

Why Use It?

EmbeddedPython executes per-document Python code inside the Lucille JVM using GraalPy. Instead of returning a JSON object, your script mutates the current document directly through a Python-friendly proxy bound as doc (and the raw Java document as rawDoc). This avoids ports, subprocesses, venvs, and per-document JSON round trips.

When To Use It

Use EmbeddedPython when you need one or more of the following:

Minimal operational overhead (ports, subprocess lifecycle, venv creation, pip installs).
No use of any external Python libraries or native dependencies that require a real Python environment.
Lightweight field enrichment/transformation.

When To Use ExternalPython Instead

Avoid EmbeddedPython and use ExternalPython when you need one or more of the following:

ExternalPython

Tue, 16 Dec 2025 00:00:00 +0000

Why Use It?

ExternalPython delegates per-document processing to an external Python process using Py4J. Lucille serializes the Document into a request, calls a Python function, receives a JSON response, and applies that response back onto the document.

When To Use It

Use ExternalPython when you need one or more of the following:

Real Python compatibility (including packages with native dependencies).
Dependency management via a requirements.txt installed into a managed venv.
Process isolation apart from the JVM.

When To Use EmbeddedPython Instead

Avoid ExternalPython and use EmbeddedPython when you need one or more of the following:

Stages on Lucille

All Stages

Field Manipulation

CopyFields

RenameFields

ChunkText

Configuration

Configuration Parameters

Chunking Methods

sentence (default)

PromptOllama

Ollama

QueryOpensearch

OpenSearch Templates

EmbeddedPython

Why Use It?

When To Use It

When To Use ExternalPython Instead

ExternalPython

Why Use It?

When To Use It

When To Use EmbeddedPython Instead

`sentence` (default)