Document on Lucille

Overview

Thu, 23 Oct 2025 00:00:00 +0000

A Document is an ordered, named set of fields. Each field may hold a single value or a list of values (multi-valued). All field values are ultimately represented in JSON. Every Document has a unique id.

Creating a Document

Use the static factory methods on the Document interface:

// Create with an explicit ID
Document doc = Document.create("my-doc-123");

// Create with an auto-generated UUID
Document doc = Document.create();

Connectors typically create Documents and call publisher.publish(doc) to send them into the pipeline.

Document IDs

Mon, 09 Jun 2025 00:00:00 +0000

Why IDs Must Be Deterministic

In search ingestion, the document ID serves as the primary key in the search index. When you index a document with ID “ABC”, the search engine either creates a new record or updates the existing record with that ID (upsert semantics). This has a critical implication:

If you run the same ingest twice, you want the same documents to get the same IDs. Otherwise, the second run creates duplicates instead of updating existing records. A search index with 100,000 documents would become 200,000 documents after a re-run — all duplicates.

Document Model

Mon, 09 Jun 2025 00:00:00 +0000

Why the Document API Matters

At first glance, the idea of a “document” in a search ingestion system seems simple: it’s a bag of named fields with values. A Map<String, Object> would seem to suffice. In practice, a well-designed Document API turns out to be one of the most important practical considerations in a search ETL framework, for reasons that only become apparent when you’ve written dozens of pipeline stages and dealt with the realities of search engine field models.