Stages

• 1: All Stages
• 2: ChunkText
• 3: PromptOllama
• 4: QueryOpensearch
• 5: EmbeddedPython
• 6: ExternalPython

{
  name: "AddRandomBoolean-First"
  class: "com.kmwllc.lucille.stage.AddRandomBoolean"
  field_name: "rand_bool_1"
  percent_true: 65
}

{
  class: "com.kmwllc.lucille.stage.MyStage"
  conditions: [
    { fields: ["content"] }
  ]
}

{
  name: "print-1"
  class: "com.kmwllc.lucille.stage.Print"
  conditions: [
    { fields: ["city"], values: ["Boston", "New York"] }
  ]
}

{
  class: "com.kmwllc.lucille.stage.OpenAIEmbed"
  conditions: [
    { fields: ["embedding"], operator: "must_not" }
  ]
}

{
  class: "com.kmwllc.lucille.stage.OpenAIEmbed"
  conditionPolicy: "all"
  conditions: [
    { fields: ["content"] }
    { fields: ["content_type"], values: ["article"] }
  ]
}

{
  class: "com.kmwllc.lucille.stage.DropDocument"
  conditions: [
    { fields: ["category"], valuesPath: "s3://my-bucket/excluded-categories.txt" }
  ]
}

1 - All Stages

This page lists all Stages available in lucille-core and as optional plugin modules.

All stages share common configuration parameters:

Field Manipulation

CopyFields

com.kmwllc.lucille.stage.CopyFields

Copies one or more source fields to destination fields.

{ class: "com.kmwllc.lucille.stage.CopyFields", source: ["title"], dest: ["title_copy"] }

RenameFields

com.kmwllc.lucille.stage.RenameFields

Renames fields by mapping old names to new names.

{ class: "com.kmwllc.lucille.stage.RenameFields", fieldMapping: { old_field: new_field } }

DeleteFields

com.kmwllc.lucille.stage.DeleteFields

Removes specified fields from a Document.

SetStaticValues

com.kmwllc.lucille.stage.SetStaticValues

Sets one or more fields to fixed, static values.

{ class: "com.kmwllc.lucille.stage.SetStaticValues", fields: { source: "my-connector", version: 2 } }

Concatenate

com.kmwllc.lucille.stage.Concatenate

Concatenates the values of multiple source fields into a destination field.

SplitFieldValues

com.kmwllc.lucille.stage.SplitFieldValues

Splits a field value (or multi-valued field) by a delimiter into a list.

RemoveDuplicateValues

com.kmwllc.lucille.stage.RemoveDuplicateValues

Removes duplicate values from multi-valued fields.

RemoveEmptyFields

com.kmwllc.lucille.stage.RemoveEmptyFields

Removes fields whose value is null, an empty string, or an empty list.

NormalizeFieldNames

com.kmwllc.lucille.stage.NormalizeFieldNames

Normalizes field names (e.g., lowercasing, replacing spaces with underscores).

DropValues

com.kmwllc.lucille.stage.DropValues

Removes specific values from multi-valued fields.

Text Processing

TrimWhitespace

com.kmwllc.lucille.stage.TrimWhitespace

Trims leading and trailing whitespace from string fields.

RemoveDiacritics

com.kmwllc.lucille.stage.RemoveDiacritics

Normalizes accented characters to their ASCII equivalents (e.g., é → e).

TruncateField

com.kmwllc.lucille.stage.TruncateField

Truncates string field values to a maximum length.

Length

com.kmwllc.lucille.stage.Length

Computes the length (number of characters or list elements) of field values and stores the result.

ApplyRegex

com.kmwllc.lucille.stage.ApplyRegex

Applies a regular expression to one or more fields. Can extract capture groups or check for matches.

ReplacePatterns

com.kmwllc.lucille.stage.ReplacePatterns

Performs pattern-based find-and-replace on string field values.

Base64Decode

com.kmwllc.lucille.stage.Base64Decode

Decodes a Base64-encoded field value into a byte array or string.

NormalizeText

com.kmwllc.lucille.stage.NormalizeText

Applies Unicode normalization and optional case folding to text fields.

ExtractFirstCharacter

com.kmwllc.lucille.stage.ExtractFirstCharacter

Extracts the first character of a string field.

CreateStaticTeaser

com.kmwllc.lucille.stage.CreateStaticTeaser

Generates a teaser (short excerpt) from a text field.

HashFieldValueToBucket

com.kmwllc.lucille.stage.HashFieldValueToBucket

Hashes a field’s value and assigns the document to a numbered bucket (useful for deterministic partitioning).

Type Conversion & Parsing

ParseDate

com.kmwllc.lucille.stage.ParseDate

Parses date strings using configurable format patterns and writes an Instant or formatted string.

ParseFloats

com.kmwllc.lucille.stage.ParseFloats

Parses a JSON array string (e.g., "[0.1, 0.2, 0.3]") into a list of floats.

ParseFilePath

com.kmwllc.lucille.stage.ParseFilePath

Extracts path components (directory, filename, extension) from a file path field.

ParseJson

com.kmwllc.lucille.stage.ParseJson

Parses a JSON string field into a JsonNode.

Timestamp

com.kmwllc.lucille.stage.Timestamp

Writes the current timestamp to a field.

Document Flow Control

DropDocument

com.kmwllc.lucille.stage.DropDocument

Marks a document as dropped. It will not be sent to the Indexer.

Typically used with conditions to selectively drop documents:

{
  class: "com.kmwllc.lucille.stage.DropDocument"
  conditions: [
    { fields: ["status"], values: ["deleted"] }
  ]
}

SkipDocument

com.kmwllc.lucille.stage.SkipDocument

Marks a document as skipped. It bypasses all downstream Stages but still reaches the Indexer. Used to issue deletes against a search backend.

{
  class: "com.kmwllc.lucille.stage.SkipDocument"
  conditions: [
    { fields: ["is_deleted"], values: ["true"] }
  ]
}

Contains

com.kmwllc.lucille.stage.Contains

Checks whether a field’s value is contained in a configured list. Sets a boolean result field.

EmitNestedChildren

com.kmwllc.lucille.stage.EmitNestedChildren

Extracts a nested array from a Document and emits each array element as an independent child Document.

CreateChildrenStage

com.kmwllc.lucille.stage.CreateChildrenStage

Generates child documents from the current document’s fields using configurable rules.

CollapseChildrenDocuments

com.kmwllc.lucille.stage.CollapseChildrenDocuments

Merges child document field values back onto the parent document.

HTML, XML, and Web

ApplyJSoup

com.kmwllc.lucille.stage.ApplyJSoup

Parses HTML content using JSoup and extracts text or attribute values using CSS selectors.

Each destination field definition requires:

• type: "text" (inner text) or "attr" (attribute value).
• selector: CSS selector.
• attr: (if type is "attr") the attribute name to extract.

{
  class: "com.kmwllc.lucille.stage.ApplyJSoup"
  byteArrayField: "html_content"
  destinationFields: {
    title: { type: "text", selector: "h1" }
    body:  { type: "text", selector: ".article-body p" }
  }
}

XPathExtractor

com.kmwllc.lucille.stage.XPathExtractor

Evaluates XPath expressions against an XML field.

FetchUri

com.kmwllc.lucille.stage.FetchUri

Fetches the content of a URL stored in a document field and stores the response as a byte array.

File Handling

ApplyFileHandlers

com.kmwllc.lucille.stage.ApplyFileHandlers

Applies configured FileHandlers to a byte array field, generating child documents for each extracted record.

FetchFileContent

com.kmwllc.lucille.stage.FetchFileContent

Loads the content of a file path (from local filesystem or cloud storage) into a byte array field on the document.

ComputeFieldSize

com.kmwllc.lucille.stage.ComputeFieldSize

Measures the byte size of a field value (e.g., a byte array or string) and stores the result.

TextExtractor

com.kmwllc.lucille.tika.stage.TextExtractor (requires lucille-tika)

Extracts text from over 1,000 file formats — PDF, Microsoft Office documents, HTML, images with embedded OCR, and many more — using Apache Tika. Reads raw bytes from a source field and writes extracted text to a destination field.

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-tika</artifactId>
  <version>${lucille.version}</version>
</dependency>

{
  name: "extract-text"
  class: "com.kmwllc.lucille.tika.stage.TextExtractor"
  source: "file_content"
  dest: "extracted_text"
}

ApplyOCR

com.kmwllc.lucille.ocr.stage.ApplyOCR (requires lucille-ocr)

Performs optical character recognition on image fields using Tesseract. Reads image bytes from a source field and writes the recognized text to a destination field. Requires Tesseract to be installed on the system running Lucille.

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-ocr</artifactId>
  <version>${lucille.version}</version>
</dependency>

{
  name: "ocr"
  class: "com.kmwllc.lucille.ocr.stage.ApplyOCR"
  source: "image_bytes"
  dest: "ocr_text"
  language: "eng"
}

Enrichment & Lookup

DictionaryLookup

com.kmwllc.lucille.stage.DictionaryLookup

Looks up field values in a term dictionary and adds matched entries as field values.

QueryDatabase

com.kmwllc.lucille.stage.QueryDatabase

Executes a JDBC prepared statement using document field values as parameters and merges the result onto the document. Useful for per-document database enrichment (e.g., joining a lookup table for each document mid-pipeline).

ElasticsearchLookup

com.kmwllc.lucille.stage.ElasticsearchLookup

Performs a lookup against an Elasticsearch index and merges matching fields onto the document.

QueryOpensearch

com.kmwllc.lucille.stage.QueryOpensearch

Executes a search template against an OpenSearch index using document field values as parameters. See QueryOpensearch for full documentation.

MatchQuery

com.kmwllc.lucille.stage.MatchQuery

Executes a match query against a configured search backend and enriches the document with matching results.

AI / ML

OpenAIEmbed

com.kmwllc.lucille.stage.OpenAIEmbed

Generates vector embeddings for a text field using the OpenAI Embeddings API.

Supported models: text-embedding-3-small, text-embedding-3-large, text-embedding-ada-002

Text is truncated to 8,191 tokens before embedding (the OpenAI API limit). Lucille uses jtokkit for accurate token counting before the API call.

{
  class: "com.kmwllc.lucille.stage.OpenAIEmbed"
  source: "content"
  dest: "content_vector"
  modelName: "text-embedding-3-small"
  apiKey: ${OPENAI_API_KEY}
  embedDocument: true
  embedChildren: true
}

JlamaEmbed

com.kmwllc.lucille.jlama.stage.JlamaEmbed (requires lucille-jlama)

Generates vector embeddings using a quantized LLM running locally inside the JVM via Jlama. No API key or external service required — the model runs directly in the Lucille process. Useful for teams with data-residency or compliance constraints that prevent sending documents to external APIs.

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-jlama</artifactId>
  <version>${lucille.version}</version>
</dependency>

{
  name: "embed"
  class: "com.kmwllc.lucille.jlama.stage.JlamaEmbed"
  source: "content"
  dest: "content_vector"
  modelPath: "/models/my-embedding-model"
}

PromptOllama

com.kmwllc.lucille.stage.PromptOllama

Sends document fields to a locally-running Ollama LLM and merges the JSON response back onto the document. See PromptOllama for full documentation.

EmbeddedPython

com.kmwllc.lucille.stage.EmbeddedPython

Runs per-document Python code inside the JVM using GraalPy. See EmbeddedPython for full documentation.

ExternalPython

com.kmwllc.lucille.stage.ExternalPython

Delegates per-document processing to an external Python process via Py4J. See ExternalPython for full documentation.

ApplyJavascript

com.kmwllc.lucille.stage.ApplyJavascript

Runs a JavaScript snippet per document using GraalVM’s JavaScript engine.

ApplyJSONata

com.kmwllc.lucille.stage.ApplyJSONata

Applies a JSONata expression to transform the document’s JSON representation.

ExtractEntitiesFST

com.kmwllc.lucille.stage.ExtractEntitiesFST

Performs named entity recognition using a finite-state transducer dictionary.

ExtractEntities

com.kmwllc.lucille.stage.ExtractEntities

Extracts entities from text fields using configured rules.

ApplyOpenNLPNameFinders (requires lucille-entity-extraction)

com.kmwllc.lucille.entity.stage.ApplyOpenNLPNameFinders

Performs named entity recognition (NER) using Apache OpenNLP models. Identifies entities such as people, organizations, and locations in text fields.

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-entity-extraction</artifactId>
  <version>${lucille.version}</version>
</dependency>

DetectLanguage

com.kmwllc.lucille.stage.DetectLanguage

Detects the language of a text field and writes the ISO language code to a destination field.

ChunkText

com.kmwllc.lucille.stage.ChunkText

Splits a long text field into smaller chunks suitable for embedding (RAG pipelines). Each chunk is emitted as a child document. See ChunkText for full documentation.

Chunking methods:

• sentence — Detects sentence boundaries using OpenNLP.
• paragraph — Splits on consecutive line breaks (\n\n, \r\n\r\n, etc.).
• fixed — Splits every lengthToSplit characters.
• custom — Splits on occurrences of the regex pattern.

Child document fields: Each child document receives id (parent ID + chunk number), parent_id, offset, length, chunk_number, total_chunks, and the chunk content in dest.

RandomVector

com.kmwllc.lucille.stage.RandomVector

Generates a random float vector and sets it on a document field. Useful for testing vector search pipelines.

Testing & Debugging

Print

com.kmwllc.lucille.stage.Print

Logs documents in JSON format at INFO level and/or writes them to a file. Place Print anywhere in the pipeline to capture the document state at that point.

{
  class: "com.kmwllc.lucille.stage.Print"
  shouldLog: false
  outputFile: "/tmp/pipeline-output.jsonl"
}

Capture and Replay

Print enables a useful development pattern: run a pipeline to capture its output to disk, then replay that output directly into a search backend without re-running the enrichment.

Step 1 — Capture. Add Print to the end of your pipeline with an outputFile. Use a NopIndexer (or set sendEnabled: false on a real indexer) so no data is actually indexed during the capture run:

pipelines: [{
  name: my-pipeline
  stages: [
    { class: "com.kmwllc.lucille.stage.SomeExpensiveEnrichment" ... }
    {
      class: "com.kmwllc.lucille.stage.Print"
      shouldLog: false
      outputFile: "/tmp/captured.jsonl"
    }
  ]
}]
indexer { type: Nop }

Step 2 — Replay. Point a FileConnector at the captured JSONL file using the JSON file handler. Use a minimal or empty pipeline — the captured documents already contain all enriched fields:

connectors: [{
  name: replay
  class: "com.kmwllc.lucille.connector.FileConnector"
  pipeline: replay-pipeline
  paths: ["/tmp/captured.jsonl"]
  fileHandlers: { json: { idField: "id" } }
}]
pipelines: [{
  name: replay-pipeline
  stages: []
}]
indexer { type: OpenSearch }
opensearch { ... }

This lets you iterate on indexer configuration, field mappings, or search backend settings without repeating expensive enrichment (OCR, embedding generation, database lookups) on every attempt.

AddRandomString

com.kmwllc.lucille.stage.AddRandomString

Adds a random alphanumeric string to a field.

AddRandomInt

com.kmwllc.lucille.stage.AddRandomInt

Adds a random integer to a field.

AddRandomDouble

com.kmwllc.lucille.stage.AddRandomDouble

Adds a random double to a field.

AddRandomDate

com.kmwllc.lucille.stage.AddRandomDate

Adds a random date/timestamp to a field within a configured range.

AddRandomBoolean

com.kmwllc.lucille.stage.AddRandomBoolean

Adds a random boolean to a field.

AddRandomNestedField

com.kmwllc.lucille.stage.AddRandomNestedField

Adds a nested JSON object with random values to a field. Useful for testing nested document structures.

2 - ChunkText

The ChunkText Stage splits a long text field into smaller, optionally overlapping segments. Each segment is emitted as a child document that flows independently through all downstream stages and is indexed as its own record. This is the foundation of retrieval-augmented generation (RAG) pipelines in Lucille.

Configuration

{
  class: "com.kmwllc.lucille.stage.ChunkText"
  source: "body"
  dest: "text"
  chunkingMethod: "sentence"
  chunksToMerge: 5
  chunksToOverlap: 1
  cleanChunks: true
}

Configuration Parameters

Chunking Methods

sentence (default)

Detects sentence boundaries using an Apache OpenNLP sentence detector. This produces semantically coherent chunks. Best used with chunksToMerge to combine a few sentences into each final chunk.

{
  class: "com.kmwllc.lucille.stage.ChunkText"
  source: "body"
  chunkingMethod: "sentence"
  chunksToMerge: 4      # 4 sentences per chunk
  chunksToOverlap: 1    # 1 sentence of overlap between chunks
}

paragraph

Splits on consecutive line breaks (\n\n, \r\n\r\n, etc.). Suitable for documents with paragraph structure.

{
  class: "com.kmwllc.lucille.stage.ChunkText"
  source: "body"
  chunkingMethod: "paragraph"
  characterLimit: 1000    # Cap chunks at 1000 characters
}

fixed

Splits every lengthToSplit characters. Simple and predictable, but may cut mid-sentence.

{
  class: "com.kmwllc.lucille.stage.ChunkText"
  source: "body"
  chunkingMethod: "fixed"
  lengthToSplit: 512
}

custom

Splits on occurrences of a regex pattern. Useful when documents have a known structural delimiter.

{
  class: "com.kmwllc.lucille.stage.ChunkText"
  source: "body"
  chunkingMethod: "custom"
  regex: "---+"    # Split on markdown horizontal rules
}

Processing Order

The Stage applies transformations in this order:

1. Initial chunking by the chosen method.
2. Cleaning (if cleanChunks: true): strip newlines, trim whitespace.
3. Pre-merge filtering: drop or truncate chunks based on preMergeMinChunkLen / preMergeMaxChunkLen.
4. Merging (if chunksToMerge > 1): combine N chunks into each final chunk.
5. Overlap (if chunksToOverlap or overlapPercentage set): prepend content from adjacent chunks.
6. Character limiting (if characterLimit set): truncate oversized final chunks.

Child Document Fields

Each chunk becomes a child document with these fields:

All other fields from the parent document are not copied to child documents unless a downstream Stage does so explicitly.

Typical RAG Pipeline

pipelines: [
  {
    name: "rag-pipeline"
    stages: [
      # 1. Extract text from PDF bytes
      {
        class: "com.kmwllc.lucille.tika.stage.TextExtractor"
        source: "file_content"
        dest: "body"
      },
      # 2. Split into sentence-merged chunks
      {
        class: "com.kmwllc.lucille.stage.ChunkText"
        source: "body"
        dest: "chunk_text"
        chunkingMethod: "sentence"
        chunksToMerge: 5
        chunksToOverlap: 1
        cleanChunks: true
        characterLimit: 2000
      },
      # 3. Embed each chunk (only child docs, which carry the chunk text)
      {
        class: "com.kmwllc.lucille.stage.OpenAIEmbed"
        source: "chunk_text"
        dest: "chunk_vector"
        embedDocument: false
        embedChildren: true
        apiKey: ${OPENAI_API_KEY}
      }
    ]
  }
]

Tips

• Use embedDocument: false, embedChildren: true in OpenAIEmbed to only embed chunks, not the full parent document.
• Set characterLimit to stay within your embedding model’s token limit. For text-embedding-3-small, 8,191 tokens ≈ roughly 6,000–7,000 characters of English text.
• Use cleanChunks: true when the source text has formatting artifacts (extra whitespace, embedded newlines from PDF extraction).
• chunksToOverlap and chunksToMerge work together: if you merge 5 sentences with 1 overlap, each chunk shares its last sentence with the next chunk, helping the model retrieve context that spans a chunk boundary.

3 - PromptOllama

What if you could just, actually, put an LLM on everything?

Ollama

Ollama allows you to run a variety of Large Language Models (LLMs) with minimal setup. You can also create custom models using Modelfiles and system prompts.

The PromptOllama Stage allows you to connect to a running instance of Ollama Server, which communicates with an LLM through a simple API. The Stage sends part (or all) of a Document to the LLM for generic enrichment. You’ll want to create a custom model (with a Modelfile) or provide a System Prompt in the Stage Config that is tailored to your pipeline.

We strongly recommend you have the LLM output only a JSON object for two main reasons: Firstly, LLMs tend to follow instructions better when instructed to do so. Secondly, Lucille can then parse the JSON response and fully integrate it into your Document.

Example

Let’s say you are working with Documents which represent emails, and you want to monitor them for potential signs of fraud. Lucille doesn’t have a DetectFraud Stage (at time of writing), but you can use PromptOllama to add this information with an LLM.

• Modelfile: Let’s say you created a custom model, fraud_detector, in your instance of Ollama Server. As part of the modelfile, you instruct the model to check the contents for fraud and output a JSON object containing just a boolean value (under fraud). Your Stage would be configured like so:

{
  name: "Ollama-Fraud"
  class: "com.kmwllc.lucille.stage.PromptOllama"
  hostURL: "http://localhost:9200"
  modelName: "fraud_detector"
  fields: ["email_text"]
}

• System Prompt: You can also just reference a specific LLM directly, and provide a system prompt in the Stage configuration.

{
  name: "Ollama-Fraud"
  class: "com.kmwllc.lucille.stage.PromptOllama"
  hostURL: "http://localhost:9200"
  modelName: "gemma3"
  systemPrompt: """
    You are to read the text inside "email_text" and output a JSON object
    containing only one field, fraud, a boolean, representing whether the
    text contains evidence of fraud or not.
  """
  fields: "email_text"
}

Regardless of the approach you choose, the LLM will receive a request that looks like this:

{
  "email_text": "Let's be sure to juice the numbers in our next quarterly earnings report."
}

(Since fields: ["email_text"], any other fields on this Document are not part of the request.)

And the response from the LLM should look like this:

{
  "fraud": true
}

Lucille will then add all key-value pairs in this response JSON into your Document. So, the Document will become:

{
  "id": "emails.csv-85",
  "run-id": "f9538992-5900-459a-90ce-2e8e1a85695c",
  "email_text": "Let's be sure to juice the numbers in our next quarterly earnings report.",
  "fraud": true
}

As you can see, PromptOllama is very versatile, and can be used to enrich your Documents in a lot of ways.

4 - QueryOpensearch

com.kmwllc.lucille.stage.QueryOpensearch

OpenSearch Templates

You can use templates in OpenSearch to repeatedly run a certain query using different parameters. For example, if we have an index full of parks, and we want to search for a certain park, we might use a template like this:

{
  "source": {
    "query": {
      "match_phrase": {
        "park_name": "{{park_to_search}}"
      }
    }
  }
}

In Opensearch, you could then call this template (providing it park_to_search) instead of writing out the full query each time you want to search.

Templates can also have default values. For example, if you want park_to_search to default to “Central Park” when a value is not provided, it would be written as: "park_name": "{{park_to_search}}{{^park_to_search}}Central Park{{/park_to_search}}"

QueryOpensearch Stage

The QueryOpensearch Stage executes a search template using certain fields from a Document as your parameters and adding OpenSearch’s response to the Document. You’ll specify either templateName, the name of a search template you’ve saved, or searchTemplate, the template you want to execute, in your Config.

You’ll also need to specify the names of parameters in your search template. These will need to match the names of fields on your Documents. If your names don’t match, you can use the RenameFields Stage first.

In particular, you have to specify which parameters are required and which are optional. If a required name in requiredParamNames is missing from a Document, an Exception will be thrown, and the template will not be executed. If an optional name in optionalParamNames is missing they (naturally) won’t be part of the template execution, so the default value will be used by OpenSearch.

If a parameter without a default value is missing, OpenSearch doesn’t throw an Exception - it just returns an empty response with zero hits. So, it is very important that requiredParamNames and optionalParamNames are defined very carefully!

Connection and Authentication

The stage connects to OpenSearch via the same opensearch config block used by the OpenSearch Indexer. Authentication and TLS are handled by OpenSearchUtils:

• Basic auth: Embed credentials in the URL: https://username:password@host:9200
• Self-signed certs: Set acceptInvalidCert: true to skip certificate validation (development only).

{
  name: "QueryOpensearch-Example"
  class: "com.kmwllc.lucille.stage.QueryOpensearch"

  opensearch {
    url: "https://admin:admin@localhost:9200"
    index: "my-index"
    acceptInvalidCert: true
  }

  templateName: "my-saved-template"
  requiredParamNames: ["park_to_search"]
  optionalParamNames: ["city"]
  destinationField: "search_results"
  opensearchResponsePath: "/hits/hits"
}

Configuration Parameters

5 - EmbeddedPython

Why Use It?

EmbeddedPython executes per-document Python code inside the Lucille JVM using GraalPy. Instead of returning a JSON object, your script mutates the current document directly through a Python-friendly proxy bound as doc (and the raw Java document as rawDoc). This avoids ports, subprocesses, venvs, and per-document JSON round trips.

When To Use It

Use EmbeddedPython when you need one or more of the following:

• Minimal operational overhead (ports, subprocess lifecycle, venv creation, pip installs).
• No use of any external Python libraries or native dependencies that require a real Python environment.
• Lightweight field enrichment/transformation.

When To Use ExternalPython Instead

Avoid EmbeddedPython and use ExternalPython when you need one or more of the following:

• Real Python compatibility (including packages with native dependencies).
• Dependency management via a requirements.txt installed into a managed venv.
• Process isolation apart from the JVM.

Example

Input Document

{
  "id": "doc-1",
  "title": "Hello",
  "author": "Test",
  "views": 123
}

Python Script

doc["title"] = doc["title"].upper()

Output Document

{
  "id": "doc-1",
  "title": "HELLO",
  "author": "Test",
  "views": 123
}

Config Parameters

{
 name: "EmbeddedPython-Example"
 class: "com.kmwllc.lucille.stage.EmbeddedPython"

 # Specify exactly one of the following:
 script_path: "/path/to/my_script.py"
 script: "doc['title'] = doc['title'].upper()"
}

script_path can begin with classpath: to load the script as a classpath resource rather than a filesystem path:

script_path: "classpath:scripts/my_script.py"

This lets you package Python scripts inside your JAR as resources (under src/main/resources/). The Lucille process loads them directly from the classpath at runtime, so you don’t need to place files in specific filesystem locations or ensure the process has read access to them.

6 - ExternalPython

Why Use It?

ExternalPython delegates per-document processing to an external Python process using Py4J. Lucille serializes the Document into a request, calls a Python function, receives a JSON response, and applies that response back onto the document.

When To Use It

Use ExternalPython when you need one or more of the following:

• Real Python compatibility (including packages with native dependencies).
• Dependency management via a requirements.txt installed into a managed venv.
• Process isolation apart from the JVM.

When To Use EmbeddedPython Instead

Avoid ExternalPython and use EmbeddedPython when you need one or more of the following:

• Minimal operational overhead (ports, subprocess lifecycle, venv creation, pip installs).
• No use of any external Python libraries or native dependencies that require a real Python environment.
• Lightweight field enrichment/transformation.

Restrictions

Your python file must be in one of the following directories that start in the current working directory that is running lucille:

• ./python
• ./src/main/resources
• ./src/test/resources
• ./src/test/resources/ExternalPythonTest (for testing)

Example

Input Document

{
  "id": "doc-1",
  "title": "Hello",
  "author": "Test",
  "views": 123
}

Python Script

def process_document(doc):
    title = doc["title"]
  
    return {
        "title": title.upper()
    }

Python Returns

{
  "title": "HELLO"
}

Output Document

{
  "id": "doc-1",
  "title": "HELLO"
}

Config Parameters

{
  name: "ExternalPython-Example"
  class: "com.kmwllc.lucille.stage.ExternalPython"

  scriptPath: "/path/to/my_script.py"

  # Optional
  pythonExecutable: "python3"
  requirementsPath: "/path/to/requirements.txt"
  functionName: "process_document"
  port: 25333
}

Example (NumPy)

Input Document

{
  "id": "doc-2",
  "values": [1, 2, 3, 4, 5]
}

Python Script

import numpy as np

def process_document(doc):
    arr = np.array(doc["values"], dtype=float)
  
    return {
        "values": doc["values"],
        "mean": float(np.mean(arr)),
        "stddev": float(np.std(arr))
    }

Output Document

{
  "id": "doc-2",
  "values": [1, 2, 3, 4, 5],
  "mean": 3.0,
  "stddev": 1.41
}

requirements.txt

numpy

Config Parameters

{
  name: "ExternalPython-Numpy"
  class: "com.kmwllc.lucille.stage.ExternalPython"

  scriptPath: "/path/to/my_numpy_script.py"
  requirementsPath: "/path/to/requirements.txt"
  
  # Optional
  pythonExecutable: "python3"
  functionName: "process_document"
  port: 25333
}