Indexers

• 1: Solr Indexer
• 2: OpenSearch Indexer
• 3: Elasticsearch Indexer
• 4: CSV Indexer
• 5: NopIndexer (No-op)
• 6: Pinecone Indexer
• 7: Weaviate Indexer

indexer {
  type: "Solr"
  batchSize: 100
  batchTimeout: 100
  blacklist: ["internal_field"]
}

solr {
  url: "http://localhost:8983/solr/my-collection"
}

indexer {
  type: "Solr"
  deletionMarkerField: "file_expired"
  deletionMarkerFieldValue: "true"
  deleteByFieldField: "delete_by_field"
  deleteByFieldValue: "delete_by_value"
}

1 - Solr Indexer

com.kmwllc.lucille.indexer.SolrIndexer

Config block: solr { ... }

*Provide either url or (for SolrCloud) zkHosts.

# Single node
solr { url: "http://localhost:8983/solr/my-collection" }

# SolrCloud with URL
solr {
  useCloudClient: true
  url: ["http://localhost:8983/solr"]
  defaultCollection: "my-collection"
}

# SolrCloud with ZooKeeper
solr {
  useCloudClient: true
  zkHosts: ["zk1:2181", "zk2:2181"]
  zkChroot: "/solr"
  defaultCollection: "my-collection"
}

Child Document Support

SolrIndexer is the only Lucille indexer that fully supports attached child documents. When a document has children (via doc.addChild()), they are automatically converted to Solr’s _childDocuments_ structure. The ___children field itself is never sent to Solr — only the nested Solr child documents are added.

Children respect the configured blacklist and whitelist: filtered fields are removed from child documents before they are added to the Solr parent.

Collection Routing

When indexer.indexOverrideField is set, documents are routed to different Solr collections within the same batch. The value of the override field on each document determines which collection receives it. The override field itself is stripped from the document before sending.

The target collection must already exist in Solr — the indexer does not create collections.

Interleaved Add/Delete Ordering

SolrIndexer preserves the order of adds and deletes within a batch. If a batch contains an add followed by a delete for the same document ID (or vice versa), the operations are sent to Solr in the correct sequence. This is important for incremental ingestion where a document may be re-added and then deleted (or deleted and re-added) within the same run.

Internally, the indexer detects when an add and a delete for the same ID appear in the same batch and flushes the earlier operation before queuing the later one.

Delete-by-Query via Terms Queries

When deleteByFieldField and deleteByFieldValue are configured, SolrIndexer constructs a Solr terms query for efficient bulk deletion rather than issuing individual delete-by-query calls. Multiple deletions targeting the same field are combined into a single terms query:

(+{!terms f='category' v='obsolete,deprecated'})

This reduces the number of round-trips to Solr when many documents are marked for deletion in the same batch.

Limitations

• Object fields not supported — If a document field contains a nested Map or Object, SolrIndexer throws an IndexerException. Flatten nested structures in your pipeline before indexing to Solr.

Troubleshooting

Connection validation fails:

• Single-node mode (useCloudClient: false): Validation uses solrClient.ping(). If this fails, check that the URL includes the collection path (e.g., http://localhost:8983/solr/my-collection) and that the collection exists.
• SolrCloud mode (useCloudClient: true): Validation checks cluster status via the Collections API. If this fails, the cluster is unreachable — check ZooKeeper connectivity and that at least one Solr node is running.

“Object field is not supported by the SolrIndexer”: A document field contains a nested JSON object. Add a stage to your pipeline that flattens or removes the field before it reaches the indexer.

2 - OpenSearch Indexer

com.kmwllc.lucille.indexer.OpenSearchIndexer

Config block: opensearch { ... }

Also supports indexer.routingField and indexer.versionType (via the generic indexer block).

opensearch {
  url: "https://admin:admin@localhost:9200"
  url: ${?OPENSEARCH_URL}
  index: "my-index"
  acceptInvalidCert: true
}

Routing

When indexer.routingField is set, the value of that field on each document is used as the _routing parameter in the bulk request. This controls which shard receives the document. Use this when your OpenSearch index has custom shard routing configured.

Version Type

When indexer.versionType is set to external or external_gte, the Kafka message offset is used as the document’s version number. This enables optimistic concurrency control in streaming mode — OpenSearch will reject a write if the incoming version is not greater than (or equal to, for external_gte) the existing version.

This only works with documents that carry Kafka metadata (i.e., in distributed mode where documents are KafkaDocument instances).

Partial Update Mode

When update: true, documents are sent as partial updates (doc-as-upsert) rather than full index operations:

• Full index (default): Replaces the entire document in the index. Fields not present in the new document are removed.
• Partial update: Merges fields into the existing document. Fields not present in the update are left unchanged.

opensearch {
  url: ${OPENSEARCH_URL}
  index: "my-index"
  update: true
}

Index Override

OpenSearch supports indexer.indexOverrideField for routing documents to different indices within the same batch. The value of the override field on each document determines the target index. The override field itself is stripped from the document before sending.

Retry Behavior

OpenSearchIndexer wraps both transport-level failures (connection refused, timeout) and HTTP-level failures (e.g., 429, 503) as IndexerRetryableException. This enables the base class retry machinery when indexer.maxRetries is configured.

Per-document failures in the bulk response are also wrapped with the item’s HTTP status code, allowing the retry logic to distinguish between retryable failures (e.g., 429 Too Many Requests) and permanent failures (e.g., 400 Bad Request).

Child Documents

Attached children are flattened into the parent document as nested objects. They are not indexed as separate OpenSearch documents. This is different from Solr’s _childDocuments_ approach — in OpenSearch, children become part of the parent’s JSON structure.

Troubleshooting

“index not found”: The target index must exist before indexing. Create it manually or via an index template.

TLS certificate errors: Set acceptInvalidCert: true for development environments with self-signed certificates. Do not use this in production.

Authentication: Include credentials in the URL: https://user:password@host:9200. Use environment variable substitution to avoid hardcoding: url: ${OPENSEARCH_URL}.

3 - Elasticsearch Indexer

com.kmwllc.lucille.indexer.ElasticsearchIndexer

Config block: elastic { ... }

Supports all OpenSearch parameters, plus parent-child join support:

Join field support (for parent-child mappings):

elastic {
  url: "http://localhost:9200"
  index: "my-index"
  join: {
    joinFieldName: "my_join_field"
    isChild: true
    childName: "my_child"
    parentDocumentIdSource: "parent_id_field"
  }
}
indexer {
  type: "Elasticsearch"
  routingField: "routing_field"
}

Join Field Support (Detailed)

Elasticsearch’s join field allows parent-child relationships within a single index. The ElasticsearchIndexer supports this via the elastic.join config block:

When isChild: true, the indexer adds a join field to each document with the child relation name and sets routing to the parent’s ID. Elasticsearch requires parent and child documents to be on the same shard, so indexer.routingField should also be set to the same field as parentDocumentIdSource.

Routing and Versioning

Same as OpenSearch: supports indexer.routingField for custom shard routing and indexer.versionType for optimistic concurrency control using Kafka offsets in distributed mode.

Partial Update Mode

When update: true, documents are sent as partial updates (doc-as-upsert) rather than full index operations. Same behavior as the OpenSearch Indexer.

Differences from OpenSearch Indexer

• No retry support — ElasticsearchIndexer does not wrap failures as IndexerRetryableException. The base class retry machinery will not trigger for Elasticsearch failures. If retries are needed, configure them at the Elasticsearch client or load balancer level.
• No indexOverrideField support — All documents are sent to the single configured index. You cannot route documents to different indices within the same batch.
• Child documents — The code iterates attached children but does not currently add them to the indexed document (this is a known TODO). Use emitted children (separate documents) instead of attached children if you need child documents indexed in Elasticsearch.

Troubleshooting

Join field errors: Ensure your Elasticsearch index mapping includes the join field with the correct parent and child relation names. The joinFieldName in config must match the mapping exactly.

Routing errors with joins: Parent and child documents must be on the same shard. Set indexer.routingField to the field containing the parent ID.

“index not found”: The target index must exist before indexing. Create it manually or via an index template.

4 - CSV Indexer

com.kmwllc.lucille.indexer.CSVIndexer

Config block: csv { ... }

indexer { type: "CSV" }
csv {
  path: "./output.csv"
  columns: ["id", "title", "body", "published_at"]
}

Limitations: CSVIndexer does not support indexer.indexOverrideField.

Column Ordering and Field Selection

The columns list determines both which fields are written and their column order. Fields not listed in columns are silently omitted from the output. The document’s id field is not automatically included — add it to columns explicitly if you want it in the CSV.

Multi-Valued Fields

When a document field is multi-valued, CSVIndexer writes the list’s toString() representation (e.g., [val1, val2]). This is a lossy representation — the square brackets and commas become part of the string. If you need a different delimiter or format for multi-valued fields, flatten them in a pipeline stage before indexing.

Append Mode

When append: true, the file is opened in append mode. Set includeHeader: false when appending to avoid duplicate header rows:

csv {
  path: "./output.csv"
  columns: ["id", "title", "body"]
  append: true
  includeHeader: false
}

Deletion Not Supported

CSVIndexer logs a warning if deletionMarkerField or deleteByFieldField are configured but does not perform any deletion. Documents marked for deletion are written as regular rows.

Directory Creation

CSVIndexer creates parent directories automatically if they don’t exist. You can specify a path like ./output/results/data.csv without creating the directories first.

Use Cases

CSVIndexer is primarily useful for:

• Testing — Verify pipeline output without a search backend.
• Debugging — Inspect what fields and values reach the indexer after pipeline processing.
• Exporting — Produce a file for import into another system.

It is not intended for production search indexing.

5 - NopIndexer (No-op)

com.kmwllc.lucille.indexer.NopIndexer

Discards all documents without sending them anywhere. Useful for testing pipelines when indexing output is not needed. Equivalent to setting indexer.sendEnabled: false on any other indexer.

indexer { type: "Nop" }

When to Use

• Testing pipeline logic — Run your pipeline end-to-end without needing a search backend running.
• Validating config — Confirm that connectors, stages, and the indexer block are configured correctly before pointing at a real backend.
• Benchmarking pipeline throughput — Measure how fast your pipeline processes documents in isolation, without indexer latency as a factor.

Equivalence with sendEnabled: false

NopIndexer and indexer.sendEnabled: false on another indexer are functionally identical — both discard documents after the pipeline processes them. The difference is that NopIndexer doesn’t require configuring a backend-specific block at all:

# These are equivalent:

# Option 1: NopIndexer
indexer { type: "Nop" }

# Option 2: sendEnabled on a real indexer
indexer { type: "Solr", sendEnabled: false }
solr { url: "http://localhost:8983/solr/my-collection" }

Use in RunType.TEST

Runner.runInTestMode() does not use NopIndexer. Instead, it creates the configured indexer with bypass=true, which causes sendToIndex() to skip the actual backend call while still running the full indexer loop (batching, events, metrics). This means your test config still needs a valid indexer block — but the backend does not need to be running. If you want to avoid configuring a backend block entirely in test configs, use NopIndexer explicitly:

indexer { type: "Nop" }

6 - Pinecone Indexer

com.kmwllc.lucille.pinecone.indexer.PineconeIndexer

Indexes vector embeddings into Pinecone. Config block: pinecone { ... }

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-pinecone</artifactId>
  <version>${lucille.version}</version>
</dependency>

indexer {
  class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
  deletionMarkerField: "is_deleted"
  deletionMarkerFieldValue: "true"
}

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "my-index"
  vectorField: "content_vector"
  namespace: "default"
}

Namespace Routing

When namespaces is configured, it is a map of namespace names to embedding field names. Each document is upserted into every configured namespace using the corresponding embedding field. This enables multi-vector indexing — for example, title embeddings in one namespace and body embeddings in another:

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "my-index"
  namespaces: {
    "title-ns": "title_vector"
    "body-ns": "body_vector"
  }
}

When namespaces is not set, all documents go to the default namespace using defaultEmbeddingField.

defaultEmbeddingField vs. namespaces

Two modes are available:

• Single namespace — Set defaultEmbeddingField to the name of the vector field. All documents are upserted into the default namespace.
• Multi namespace — Set namespaces as a map of namespace → embedding field. Documents are upserted into each namespace with the corresponding vector.

At least one of these must be set when uploading documents. If both are omitted, the indexer throws at upload time.

Metadata Fields

Only fields listed in metadataFields are sent as Pinecone metadata alongside the vector. All other document fields are ignored. Metadata values are converted to strings.

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "my-index"
  defaultEmbeddingField: "content_vector"
  metadataFields: ["title", "category", "source_url"]
}

Upsert vs. Update Mode

pinecone {
  mode: "update"
  // ...
}

Batch Size Limit

Pinecone’s API limits batches to 1000 vectors or 2MB, whichever is reached first. The indexer enforces the 1000-vector limit at startup — if indexer.batchSize exceeds 1000, the indexer throws an exception and refuses to start.

Higher-dimensional vectors may hit the 2MB limit at counts lower than 1000. If this happens, the Pinecone API returns an error at runtime. Reduce indexer.batchSize accordingly.

Deletion

Deletion is by ID only (uses deletionMarkerField / deletionMarkerFieldValue). Delete-by-query is not supported.

When namespaces is configured, deletion is issued to all configured namespaces.

Troubleshooting

“Maximum batch size for Pinecone is 1000”: Reduce indexer.batchSize to 1000 or lower.

API key invalid: Verify PINECONE_API_KEY is set and the key has write access to the target index.

Index not found: The Pinecone index must be created via the Pinecone console or API before running Lucille.

Vector dimension mismatch: The vector field on each document must have the same number of dimensions as the Pinecone index was created with. A mismatch causes a runtime error from the Pinecone API.

7 - Weaviate Indexer

com.kmwllc.lucille.weaviate.indexer.WeaviateIndexer

Indexes documents into Weaviate. Config block: weaviate { ... }

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-weaviate</artifactId>
  <version>${lucille.version}</version>
</dependency>

indexer {
  class: "com.kmwllc.lucille.weaviate.indexer.WeaviateIndexer"
}

weaviate {
  apiKey: ${WEAVIATE_API_KEY}
  host: "my-cluster.weaviate.network"
  className: "Article"
  idDestinationName: "lucille_id"
  vectorField: "content_vector"
}

Note on IDs: Weaviate requires UUIDs for its internal id field. The WeaviateIndexer generates a UUID from the document’s Lucille ID and stores the original ID under idDestinationName so it can be retrieved later.

UUID Generation

Weaviate requires UUIDs for its internal id field. The WeaviateIndexer generates a deterministic UUID from the document’s Lucille ID using UUID.nameUUIDFromBytes(id.getBytes()). This means:

• The same Lucille ID always maps to the same Weaviate UUID, enabling idempotent upserts.
• The original Lucille ID is stored under idDestinationName (default: "id_original") so it can be retrieved later.

Deletion Not Supported

WeaviateIndexer logs a warning if deletionMarkerField or deleteByFieldField are configured but does not perform deletions. Documents marked for deletion are indexed as regular objects. This is a known limitation.

Vector Field Handling

If vectorField is set and the document has that field, the vector is sent alongside the object properties. The vector field is removed from the properties map to avoid storing it twice (once as a vector, once as a property).

If vectorField is omitted, no vector is sent. This is useful when Weaviate is configured to generate its own embeddings via vectorizer modules (e.g., text2vec-openai).

Consistency Level

All writes use ConsistencyLevel.ALL (strongest consistency). This is not currently configurable. In a multi-node Weaviate cluster, this means all replicas must acknowledge the write before it is considered successful.

Connection Timeouts

The Weaviate client is configured with 6-second timeouts for connection, read, and write operations. These are not currently configurable via the Lucille config. If you experience timeout errors with large batches or slow networks, reduce indexer.batchSize.

Class/Schema Requirements

The className parameter determines which Weaviate class (schema type) objects are created under. The class must already exist in the Weaviate schema — the indexer does not create it. If the class doesn’t exist, the batch write will fail.

Troubleshooting

AuthException (“Couldn’t connect to Weaviate instance”): The API key is invalid or the host is unreachable. Verify WEAVIATE_API_KEY and that the host value is correct (hostname only, no https:// prefix).

Class not found in schema: Create the Weaviate class via the Weaviate console or REST API before running Lucille.

Vector dimension mismatch: If using a Weaviate vectorizer module, ensure the vector field dimensions match what the module expects. If sending vectors directly, ensure they match the class’s configured vector dimensions.

Timeout errors: Reduce indexer.batchSize to send smaller batches. The 6-second timeout is fixed.