Document

• 1: Overview
• 2: Document IDs
• 3: Document Model

1 - Overview

A Document is an ordered, named set of fields. Each field may hold a single value or a list of values (multi-valued). All field values are ultimately represented in JSON. Every Document has a unique id.

Creating a Document

Use the static factory methods on the Document interface:

// Create with an explicit ID
Document doc = Document.create("my-doc-123");

// Create with an auto-generated UUID
Document doc = Document.create();

Connectors typically create Documents and call publisher.publish(doc) to send them into the pipeline.

Reserved Fields

Lucille reserves several field names for internal use. Do not use these for application data.

Field Types

Lucille Documents support the following value types:

• String
• Boolean
• Integer
• Double
• Float
• Long
• java.time.Instant
• byte[]
• com.fasterxml.jackson.databind.JsonNode
• java.sql.Timestamp
• java.util.Date

Reading Fields

Single-Valued Access

String  title  = doc.getString("title");
int     count  = doc.getInt("count");
double  score  = doc.getDouble("score");
float   weight = doc.getFloat("weight");
long    size   = doc.getLong("size");
boolean flag   = doc.getBoolean("active");
Instant ts     = doc.getInstant("created_at");
byte[]  raw    = doc.getBytes("content");
JsonNode json  = doc.getJson("metadata");

Multi-Valued Access

These methods also wrap a single value in a list if needed:

List<String>  titles = doc.getStringList("title");
List<Integer> counts = doc.getIntList("counts");
List<Double>  scores = doc.getDoubleList("scores");
List<Long>    sizes  = doc.getLongList("sizes");

Checking Field Existence

if (doc.has("title")) {
    String t = doc.getString("title");
}

Writing Fields

setField — Overwrite

Replaces any existing value(s) and makes the field single-valued:

doc.setField("title", "Hello World");
doc.setField("count", 42);
doc.setField("active", true);
doc.setField("score", 0.95);
doc.setField("created_at", Instant.now());

addToField — Append

Appends a value, converting the field to multi-valued if it was single-valued:

doc.addToField("tags", "search");
doc.addToField("tags", "etl");
// tags is now ["search", "etl"]

setOrAdd — Create or Append

Creates the field as single-valued if it does not exist; appends if it does:

doc.setOrAdd("tags", "search");
doc.setOrAdd("tags", "etl");

update — Controlled Write with UpdateMode

The update method accepts an UpdateMode enum that covers the three most common write patterns:

import com.kmwllc.lucille.core.UpdateMode;

// OVERWRITE: first value replaces all existing values; additional values are appended
doc.update("title", UpdateMode.OVERWRITE, "New Title");

// APPEND: all values are appended (field becomes or remains multi-valued)
doc.update("tags", UpdateMode.APPEND, "search", "etl");

// SKIP: field is left unchanged if it already has a value
doc.update("title", UpdateMode.SKIP, "Default Title");

Nested JSON

Documents support reading and writing values within nested JSON objects and arrays using dot-path notation (e.g., "metadata.author.name") or structured List<Document.Segment> paths.

Reading Nested Values

JsonNode node = doc.getNestedJson("metadata.author.name");

// Or using structured path segments
List<Document.Segment> path = Document.Segment.parse("metadata.items[2].title");
JsonNode node2 = doc.getNestedJson(path);

Writing Nested Values

ObjectMapper mapper = new ObjectMapper();
doc.setNestedJson("metadata.score", mapper.valueToTree(0.95));

Removing Nested Values

doc.removeNestedJson("metadata.tempField");

Path Segments

List<Document.Segment> segments = Document.Segment.parse("a.b[2].c");
String path = Document.Segment.stringify(segments); // "a.b[2].c"

Dropping and Skipping

Dropping removes the document from the pipeline entirely. It will not reach the Indexer.

Use the DropDocument Stage in config, or call setDropped() in stage code:

doc.setDropped(true);

Skipping causes the document to bypass all downstream Stages but still reach the Indexer. This is used for deletion markers, so the Indexer can issue a delete against the search backend.

Use the SkipDocument Stage in config, or call setSkipped() in stage code:

doc.setSkipped(true);

Child Documents

A Stage may generate child documents — additional Documents that flow through the remaining pipeline stages as independent records and are indexed alongside the parent. A Stage returns children from processDocument() as an Iterator<Document>.

Children are always emitted before the parent document, ensuring the Publisher’s accounting registers child IDs before it sees the parent’s completion event.

Iterating Fields

for (String fieldName : doc) {
    // iterate over all field names in the Document
}

Serialization

Documents serialize to and from JSON. In distributed mode, Documents flow between components as JSON bytes. The id and run_id are always included.

String json = doc.toString();

2 - Document IDs

Why IDs Must Be Deterministic

In search ingestion, the document ID serves as the primary key in the search index. When you index a document with ID “ABC”, the search engine either creates a new record or updates the existing record with that ID (upsert semantics). This has a critical implication:

If you run the same ingest twice, you want the same documents to get the same IDs. Otherwise, the second run creates duplicates instead of updating existing records. A search index with 100,000 documents would become 200,000 documents after a re-run — all duplicates.

Deterministic IDs mean:

• Re-ingestion is safe. Running the pipeline again updates existing documents rather than creating duplicates.
• Incremental updates work. A document that changes in the source system gets re-indexed under the same ID, replacing the old version.
• Deletions are possible. To delete a document from the index, you need to know its ID. If the ID was random, you’d have no way to reference it later.
• Parent-child relationships are stable. Child documents reference their parent by ID. If the parent’s ID changes on re-run, the relationship breaks.

This is why Connectors are responsible for creating documents with meaningful, stable IDs derived from the source data — a file path, a database primary key, a URL — rather than random UUIDs.

Why IDs Are Immutable in Lucille

Once a Document is created with an ID, that ID cannot be changed through the normal Document API. The id field is in the RESERVED_FIELDS set, so setField("id", ...), addToField("id", ...), and renameField("id", ...) all throw IllegalArgumentException.

This immutability exists because the ID is used as a tracking key throughout the system:

1. The Publisher registers the ID in its accounting ledger when the document is published. If the ID changed mid-pipeline, the Publisher would never receive a terminal event for the original ID — the run would hang.
2. The Worker sends CREATE events for child documents using their IDs. If a child’s ID changed after the CREATE event was sent, the Publisher’s accounting would be corrupted.
3. The Kafka message key is the document ID. Changing the ID mid-pipeline would break ordering guarantees (the document would land on a different partition after the change).
4. The Indexer uses the ID to send upserts and deletions to the search backend. The ID must be the same value that was tracked through the entire pipeline.

In short: the ID is the document’s identity across all components. Mutating it would break accounting, ordering, and idempotency simultaneously.

How Lucille Handles Duplicate IDs

Duplicate IDs — multiple documents with the same ID published in the same run — are a legitimate scenario. They arise in CDC (Change Data Capture) scenarios where a source system emits multiple updates for the same record, or when a connector reads from a source that contains duplicate entries.

Lucille handles duplicates at two levels:

At the Publisher Level: The Bag Data Structure

The Publisher’s docIdsToTrack is a Bag<String> (multiset), not a Set<String>. If two documents with ID “doc-1” are published, the Bag count for “doc-1” becomes 2. The Publisher expects to receive two separate terminal events for that ID. Each terminal event decrements the count by one. The run is not considered complete until the count reaches zero for all IDs.

// Two documents with same ID published → bag count is 2
docIdsToTrack.add("doc-1");  // count: 1
docIdsToTrack.add("doc-1");  // count: 2

// First terminal event → count drops to 1
docIdsToTrack.remove("doc-1", 1);  // count: 1

// Second terminal event → count drops to 0
docIdsToTrack.remove("doc-1", 1);  // count: 0, now removed

This means duplicate IDs don’t corrupt the accounting — each published document is tracked independently even if it shares an ID with another.

At the Publisher Level: Collapsing Mode

For connectors that emit multiple consecutive documents with the same ID (common in CDC), the Publisher supports a collapsing mode (requiresCollapsingPublisher() = true). In this mode, consecutive same-ID documents are merged into a single document via setOrAddAll() before being sent for processing:

if (previousDoc.getId().equals(document.getId())) {
    previousDoc.setOrAddAll(document);  // merge fields into one document
} else {
    sendForProcessing(previousDoc);     // different ID — send the previous one
    previousDoc = document;             // hold the new one
}

This reduces N consecutive same-ID documents to one document with multi-valued fields, which is then processed and indexed once. numReceived counts all N inputs; numPublished counts only the single merged output.

At the Indexer Level: Upsert Semantics

When two documents with the same ID reach the Indexer (either because collapsing is not enabled, or because they were non-consecutive), the search engine’s upsert semantics handle it: the second document overwrites the first in the index. The final state of the index reflects the last document indexed with that ID. Combined with Lucille’s ordering guarantees (same ID → same partition → same consumer → sequential processing), this means the final state is deterministic.

At the Indexer Level: Ordering Within a Batch

If a batch contains both an upsert and a delete for the same ID, the Indexer implementations (SolrIndexer, OpenSearchIndexer) explicitly handle ordering — flushing pending upserts before processing a delete for the same ID, or vice versa. This ensures that the operations are applied in the correct sequence even within a single batch.

The idOverrideField: Decoupling Internal ID from Index ID

Sometimes the ID used internally for tracking is not the ID you want in the search index. For example:

• A Connector might use a composite key (source + record number) for tracking uniqueness, but the search index expects a simpler ID.
• A pipeline might compute a better ID during enrichment (e.g., hashing certain fields to create a deduplication key).
• A document might need different IDs in different destination indices.

Lucille solves this with indexer.idOverrideField — a configuration option that tells the Indexer to use a different field’s value as the document’s ID when sending to the search backend, without modifying the document’s internal ID:

indexer {
  idOverrideField: "computed_id"
}

How It Works

1. The Connector creates the document with a stable internal ID (e.g., "source-record-42")
2. A pipeline Stage computes a better ID and stores it in a field (e.g., doc.setField("computed_id", "hash-abc123"))
3. The document flows through the system tracked by its internal ID ("source-record-42")
4. At indexing time, the Indexer calls getDocIdOverride(doc) which returns "hash-abc123"
5. The document is sent to the search backend with ID "hash-abc123"

What Uses Which ID

The internal ID ("source-record-42") is used for:

• Publisher accounting
• Kafka message key (ordering)
• Event tracking (CREATE, FINISH, FAIL)
• Logging and debugging

The override ID ("hash-abc123") is used only for:

• The document’s ID in the search index

This separation means the pipeline’s internal correctness guarantees (ordering, accounting, fault tolerance) are never affected by what ID appears in the search index. The override is applied at the very last moment — after all tracking is complete — as a pure presentation concern.

ID Generation Strategies: Good and Bad

How Existing Connectors Generate IDs

FileConnector: MD5 hash of the full file path.

String docId = DigestUtils.md5Hex(fullPath);
Document doc = Document.create(StorageClient.createDocId(docId, params));

The FileConnector uses the MD5 hash of the file’s full URI (e.g., s3://bucket/path/to/file.pdf) as the document ID. This is a good strategy because:

• It’s deterministic — the same file always produces the same ID
• It’s stable across re-runs — re-ingesting the same file updates rather than duplicates
• It handles special characters — file paths with spaces, unicode, or long lengths are reduced to a fixed-length hex string that’s safe for any search engine
• It’s unique — different file paths produce different hashes (collision probability is negligible)

The docIdPrefix is then prepended via StorageClient.createDocId(), producing IDs like "file-a1b2c3d4e5f6...".

DatabaseConnector: Value from a configured ID column.

String id = createDocId(rs.getString(idColumn));
Document doc = Document.create(id);

The DatabaseConnector reads the ID from a column specified in config (idField). This is the natural choice for database sources because:

• The database already has a primary key that uniquely identifies each record
• It’s deterministic and stable across re-runs
• It matches what the user expects — the search index ID corresponds to the database primary key

ChunkText: Parent ID + chunk number.

String id = parentId + "-" + (i + 1);
Document childDoc = Document.create(id);

Child documents derive their IDs from the parent ID plus a positional suffix. This ensures:

• Children have unique IDs (parent ID is unique, suffix is unique within the parent)
• The relationship to the parent is visible in the ID itself
• Re-chunking the same parent produces the same child IDs (deterministic)

Good ID Strategies

Bad ID Strategies

When Random UUIDs Are Acceptable

There is one scenario where random UUIDs are acceptable: when the index is always rebuilt from scratch (full re-index, not incremental). If you drop and recreate the index on every run, duplicate IDs from re-runs are not a concern because the old index is gone. However, this limits you to full batch mode and prevents incremental updates.

Hashing as an ID Strategy

Hashing (MD5, SHA-256) is useful when:

• The natural key is too long for a search engine ID field (some have length limits)
• The natural key contains characters that are problematic in URLs or APIs
• You want to deduplicate across sources (hash the content, not the source path)

The FileConnector’s use of DigestUtils.md5Hex(fullPath) is a good example. The tradeoff is that the ID is opaque — you can’t look at it and know which file it represents. The FileConnector mitigates this by also storing the full path in a file_path field on the document.

The docIdPrefix: Namespacing IDs by Connector

When multiple connectors feed documents into the same index, their IDs might collide. A database connector and a file connector might both produce a document with ID “1”. The docIdPrefix configuration on a connector prepends a string to all document IDs it creates:

connectors: [
  {
    name: "database"
    class: "com.kmwllc.lucille.connector.DatabaseConnector"
    docIdPrefix: "db-"
    # produces IDs like "db-1", "db-2", ...
  },
  {
    name: "files"
    class: "com.kmwllc.lucille.connector.FileConnector"
    docIdPrefix: "file-"
    # produces IDs like "file-/path/to/doc.pdf", ...
  }
]

The prefix is applied by the connector when creating documents via AbstractConnector.createDocId(id). It becomes part of the document’s immutable ID from that point forward.

Summary

3 - Document Model

Why the Document API Matters

At first glance, the idea of a “document” in a search ingestion system seems simple: it’s a bag of named fields with values. A Map<String, Object> would seem to suffice. In practice, a well-designed Document API turns out to be one of the most important practical considerations in a search ETL framework, for reasons that only become apparent when you’ve written dozens of pipeline stages and dealt with the realities of search engine field models.

Every pipeline stage reads fields, transforms them, and writes results back. A stage might be three lines of logic surrounded by ten lines of field access boilerplate — checking if a field exists, handling null, deciding whether to overwrite or append, converting types, dealing with single-valued vs. multi-valued fields. If the Document API is clumsy, that boilerplate dominates every stage you write. If the API is well-designed, stages are concise and the intent is clear.

Lucille’s Document API is the result of iterating on this problem across many real-world pipelines. Every method exists because a common pattern in search ingestion code demanded it.

Matching the Search Engine’s Field Model

Search engines (Solr, Elasticsearch, OpenSearch) have a field model that differs from a typical programming language map in important ways:

Single-valued vs. multi-valued fields. In a search engine schema, a field can hold one value or a list of values. A document might have a single title but multiple tags. This distinction matters for how the field is indexed, how it’s queried, and how it’s displayed. A Map<String, Object> does not capture this distinction — is the value a String or a List<String>? You’d need to check at every access point.

Lucille’s Document makes this explicit:

• getString("title") returns the single value (or the first value if multi-valued).
• getStringList("tags") returns all values as a list (wrapping a single value in a list if necessary).
• setField("title", "Hello") creates a single-valued field.
• addToField("tags", "search") converts the field to multi-valued if it wasn’t already.
• setOrAdd("tags", "etl") creates the field as single-valued if absent, or appends if present.

This mirrors exactly how search engines think about fields. A stage author doesn’t need to write if (value instanceof List) checks — the API handles the single/multi distinction uniformly.

The three update patterns. When a stage writes to a field, there are exactly three things it might want to do:

1. Overwrite whatever was there before.
2. Append to whatever was there (creating a multi-valued field).
3. Skip — write only if the field doesn’t already exist (don’t clobber earlier enrichment).

These three patterns appear so frequently in search ingestion code that Lucille provides them as a first-class UpdateMode enum, usable with the update() method:

doc.update("title", UpdateMode.OVERWRITE, "New Title");
doc.update("tags", UpdateMode.APPEND, "tag1", "tag2");
doc.update("summary", UpdateMode.SKIP, "Default summary");

The update() method also accepts varargs, so a stage can write multiple values in a single call. Without this, every stage would implement its own if/else logic for these three cases — and get it subtly wrong in edge cases (e.g., forgetting to convert a single-valued field to multi-valued before appending).

Why JSON Backing (ObjectNode) Is the Right Choice

Lucille’s Document is implemented as a thin wrapper around a Jackson ObjectNode. This is not an obvious choice — a HashMap<String, Object> would be simpler to implement. The JSON backing is motivated by the realities of how documents flow through the system.

Serialization Without Type Metadata

Documents cross boundaries constantly in Lucille: they’re placed on queues (in-memory or Kafka), sent to search backends via bulk APIs, logged for debugging, and captured in test mode for assertions. Every boundary crossing requires serialization.

With a JSON-backed document, serialization is trivial: data.toString() produces valid JSON. No conversion step, no schema, no type registry.

Critically, JSON’s type system eliminates the need to store explicit type information with each field. Jackson’s ObjectNode stores values as typed nodes — TextNode, IntNode, BooleanNode, ArrayNode, etc. When serialized to JSON, the types are implicit in the syntax:

• "title": "Hello" — string (quoted)
• "count": 42 — integer (unquoted number)
• "active": true — boolean (literal)
• "tags": ["a", "b"] — array (brackets)

When deserialized, Jackson reconstructs the correct node types from the JSON syntax. No type annotations, no class names in the serialized form, no versioning concerns. Compare this to Java serialization or a HashMap-based approach where you’d need to store type discriminators alongside values to reconstruct them correctly on the other end.

This matters operationally: documents on Kafka topics are human-readable JSON. An administrator can inspect them with standard Kafka tooling and understand what they contain without a decoder ring.

Zero-Cost Boundary Crossings

Because the document is JSON internally, there is no impedance mismatch at any boundary:

• From a JSON source (file, HTTP response, Kafka message): Parse the JSON into an ObjectNode and it becomes the Document’s backing store directly. No field-by-field copying.
• To a search engine: The bulk API for Elasticsearch, OpenSearch, and Solr all accept JSON. The document is already in the right format.
• To Kafka: Serialize the ObjectNode to a string. Done.
• From Kafka: Parse the string back to an ObjectNode. Done.
• In test assertions: document.toString() gives you the complete state as readable JSON.

A HashMap-backed document would require a serialization pass at every one of these boundaries — and in a system where a document crosses 4+ boundaries (source → processing queue → worker → indexing queue → indexer → search backend), that overhead is significant.

Native Nested Structure Support

Modern search ingestion often involves complex nested data: JSON responses from HTTP enrichment stages, structured extraction results from LLMs, nested document schemas in Elasticsearch. An ObjectNode naturally represents nested JSON (objects within objects, arrays of objects).

Lucille’s getNestedJson/setNestedJson API works directly on the tree structure:

// Read a nested value
JsonNode author = doc.getNestedJson("metadata.author.name");

// Set a nested value (creates intermediate objects as needed)
doc.setNestedJson("metadata.source.url", TextNode.valueOf("https://..."));

// Array indexing
JsonNode thirdTag = doc.getNestedJson("results[0].metadata.tags[2]");

With a HashMap, nested access would require ((Map) ((Map) map.get("metadata")).get("author")).get("name") — casting at every level, null-checking at every level, and no type safety. The Jackson tree API provides a typed, null-safe traversal.

Interop with the Jackson Ecosystem

Stages that call external APIs (HTTP enrichment, LLM calls, search engine queries) typically use HTTP clients that return Jackson JsonNode objects. These can be stored directly on the Document without conversion:

JsonNode apiResponse = httpClient.get(url);  // returns JsonNode
doc.setField("enrichment_result", apiResponse);  // stored directly

Similarly, Lucille’s JSONata transformation support operates on the Jackson tree natively — the Document’s backing ObjectNode is the input to the JSONata expression, and the result is written back as a JsonNode.

The Tradeoffs

The JSON backing is not free:

Per-field access overhead. Getting a String from an ObjectNode means node.get("field").asText() rather than (String) map.get("field"). The Document API hides this behind typed getters, but the implementation does more work per access than a HashMap. For stages that read many fields in a tight loop, this is measurably slower.

Memory overhead. Each field value is wrapped in a JsonNode subclass (TextNode, IntNode, etc.) rather than stored as a raw Java object. For documents with many small fields, the per-field wrapper overhead adds up.

No arbitrary Java types. A HashMap can store any Java object. An ObjectNode can only store JSON-representable types. Lucille works around this (byte arrays are stored as base64-encoded binary nodes, Instants are stored as ISO-8601 strings), but the Document cannot natively hold arbitrary domain objects.

Deep copy cost. Copying a document requires objectNode.deepCopy(), which recursively copies the entire JSON tree. A HashMap with immutable String values would be cheaper to shallow-copy.

In practice, these tradeoffs are acceptable because pipeline stages typically read a small number of fields, do expensive work (API calls, model inference, text processing), and write a small number of fields. The per-access overhead is negligible relative to the actual enrichment work. The serialization savings at every boundary crossing more than compensate.

Notable Design Choices in the API

Uniform Single/Multi-Valued Access

The getStringList() method returns a List<String> regardless of whether the field is single-valued or multi-valued. If the field is single-valued, it wraps the value in a singleton list. This means a stage that processes “all values of a field” doesn’t need to check whether the field is single or multi-valued first — it can always iterate over the list.

Conversely, getString() always returns the first value, whether the field is single or multi-valued. A stage that only cares about the primary value doesn’t need to handle the list case.

The setOrAdd Pattern

setOrAdd() is a single method that handles the most common field-writing pattern in search ingestion: “if this field doesn’t exist yet, create it; if it does, append to it.” Without this, every stage that accumulates values would need:

if (doc.has("tags")) {
    doc.addToField("tags", newTag);
} else {
    doc.setField("tags", newTag);
}

With setOrAdd, it’s one line: doc.setOrAdd("tags", newTag). Across a pipeline with dozens of stages, this eliminates hundreds of lines of boilerplate.

Typed Getters with Null Semantics

Every getter returns null for both “field absent” and “field present but null.” The has() method distinguishes between these cases when it matters. This is a deliberate choice: most stage logic doesn’t care why a value is missing — it just needs to handle the missing case. The rare stage that needs to distinguish “field not set” from “field explicitly null” can use has() + hasNonNull().

Reserved Fields with Triple-Underscore Prefix

Internal control fields (___dropped, ___skipped, ___children) use a triple-underscore prefix. This keeps them out of the normal field namespace — a triple-underscore prefix is highly unlikely to collide with any user-defined field name — and makes them visually distinct from user data fields. The validateFieldNames() method prevents stages from accidentally writing to reserved fields by checking membership in RESERVED_FIELDS.

Field Name Validation on Every Write

Every setter calls validateFieldNames() before writing. This catches two classes of bugs immediately:

1. Attempting to write to a reserved field (like id or run_id).
2. Passing a null or empty field name.

The validation happens at write time, not at indexing time, so bugs are caught in the stage that caused them rather than surfacing later in the pipeline.

Insertion Order Preservation

ObjectNode uses a LinkedHashMap internally, so getFieldNames() returns fields in insertion order. This is a subtle but useful property: when a document is serialized to JSON, the fields appear in the order they were added. This makes debugging easier (the ID is always first, enrichment fields appear in pipeline order) and produces deterministic output for testing.

JSONata Integration

The transform() method applies a JSONata expression directly to the Document’s backing ObjectNode. JSONata is a query and transformation language for JSON — think of it as XPath/XSLT for JSON. Because the Document is already JSON, there’s no conversion step. A stage can reshape a document’s structure with a single expression:

Jsonata expr = Jsonata.jsonata("{ 'fullName': firstName & ' ' & lastName }");
doc.transform(expr);

This is particularly powerful for stages that need to restructure complex nested data without writing procedural Java code.

The asMap() Escape Hatch

asMap() converts the Document to a Map<String, Object> using Jackson’s MAPPER.convertValue(). This is the escape hatch for code that needs a plain Map — typically when interfacing with libraries that expect Map input. It’s deliberately not the primary API because it loses the typed access, the single/multi-valued distinction, and the zero-cost serialization. But it exists for interop.

Summary

Lucille’s Document API is designed around three principles:

1. Match the search engine’s field model — single/multi-valued distinction, typed access, update modes that reflect how search fields are actually populated.
2. Minimize serialization cost — JSON backing means zero-cost boundary crossings in a system where documents cross many boundaries.
3. Eliminate stage boilerplate — setOrAdd, update with UpdateMode, uniform list access, and null handling reduce the per-stage code that isn’t core logic.

The result is that a typical pipeline stage is a few lines of domain logic rather than a page of field-access ceremony. Across a pipeline with dozens of stages, this compounds into significantly less code, fewer bugs, and faster development.