Indexers on Lucille

Solr Indexer

Mon, 09 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.indexer.SolrIndexer

Config block: solr { ... }

Parameter	Type	Required	Description
`url`	List<String>	No*	Solr base URLs (e.g., `http://localhost:8983/solr`).
`useCloudClient`	Boolean	No	Use SolrCloud client. Default: `false`.
`defaultCollection`	String	No	Target Solr collection.
`zkHosts`	List<String>	No*	ZooKeeper addresses for SolrCloud (e.g., `["zk1:2181", "zk2:2181"]`).
`zkChroot`	String	No	ZooKeeper chroot path for SolrCloud (e.g., `"/solr"`).
`userName`	String	No	HTTP Basic Auth username.
`password`	String	No	HTTP Basic Auth password.
`acceptInvalidCert`	Boolean	No	Accept invalid TLS certificates. Default: `false`.

*Provide either url or (for SolrCloud) zkHosts.

OpenSearch Indexer

Mon, 09 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.indexer.OpenSearchIndexer

Config block: opensearch { ... }

Parameter	Type	Required	Description
`url`	String	Yes	OpenSearch endpoint URL including credentials if needed (e.g., `https://admin:password@localhost:9200`).
`index`	String	Yes	Target index name.
`update`	Boolean	No	Use the partial update API instead of index (upsert). Default: `false`.
`acceptInvalidCert`	Boolean	No	Accept invalid TLS certificates. Default: `false`.

Also supports indexer.routingField and indexer.versionType (via the generic indexer block).

opensearch {
 url: "https://admin:admin@localhost:9200"
 url: ${?OPENSEARCH_URL}
 index: "my-index"
 acceptInvalidCert: true
}

Routing

When indexer.routingField is set, the value of that field on each document is used as the _routing parameter in the bulk request. This controls which shard receives the document. Use this when your OpenSearch index has custom shard routing configured.

Elasticsearch Indexer

Mon, 09 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.indexer.ElasticsearchIndexer

Config block: elastic { ... }

Supports all OpenSearch parameters, plus parent-child join support:

Parameter	Type	Required	Description
`url`	String	Yes	Elasticsearch endpoint URL.
`index`	String	Yes	Target index name.
`update`	Boolean	No	Use partial update API. Default: `false`.
`acceptInvalidCert`	Boolean	No	Accept invalid TLS certs. Default: `false`.
`parentName`	String	No	Parent relation name for join field mappings.

Join field support (for parent-child mappings):

elastic {
 url: "http://localhost:9200"
 index: "my-index"
 join: {
 joinFieldName: "my_join_field"
 isChild: true
 childName: "my_child"
 parentDocumentIdSource: "parent_id_field"
 }
}
indexer {
 type: "Elasticsearch"
 routingField: "routing_field"
}

Join Field Support (Detailed)

Elasticsearch’s join field allows parent-child relationships within a single index. The ElasticsearchIndexer supports this via the elastic.join config block:

CSV Indexer

Mon, 09 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.indexer.CSVIndexer

Config block: csv { ... }

Parameter	Type	Required	Description
`path`	String	Yes	Output CSV file path.
`columns`	List<String>	Yes	Ordered list of document fields to write as columns.
`includeHeader`	Boolean	No	Write a header row. Default: `true`.
`append`	Boolean	No	Append to an existing file. Default: `false`.

indexer { type: "CSV" }
csv {
 path: "./output.csv"
 columns: ["id", "title", "body", "published_at"]
}

Limitations: CSVIndexer does not support indexer.indexOverrideField.

NopIndexer (No-op)

Mon, 09 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.indexer.NopIndexer

Discards all documents without sending them anywhere. Useful for testing pipelines when indexing output is not needed. Equivalent to setting indexer.sendEnabled: false on any other indexer.

indexer { type: "Nop" }

When to Use

Testing pipeline logic — Run your pipeline end-to-end without needing a search backend running.
Validating config — Confirm that connectors, stages, and the indexer block are configured correctly before pointing at a real backend.
Benchmarking pipeline throughput — Measure how fast your pipeline processes documents in isolation, without indexer latency as a factor.

Equivalence with sendEnabled: false

NopIndexer and indexer.sendEnabled: false on another indexer are functionally identical — both discard documents after the pipeline processes them. The difference is that NopIndexer doesn’t require configuring a backend-specific block at all:

Pinecone Indexer

Mon, 09 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.pinecone.indexer.PineconeIndexer

Indexes vector embeddings into Pinecone. Config block: pinecone { ... }

Maven dependency:

<dependency>
 <groupId>com.kmwllc</groupId>
 <artifactId>lucille-pinecone</artifactId>
 <version>${lucille.version}</version>
</dependency>

Parameter	Type	Required	Description
`apiKey`	String	Yes	Pinecone API key. Use `${PINECONE_API_KEY}` for environment variable substitution.
`index`	String	Yes	Name of the Pinecone index to write to.
`vectorField`	String	Yes	Document field containing the vector embedding.
`namespace`	String	No	Pinecone namespace. Default: `"default"`.

indexer {
 class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
 deletionMarkerField: "is_deleted"
 deletionMarkerFieldValue: "true"
}

pinecone {
 apiKey: ${PINECONE_API_KEY}
 index: "my-index"
 vectorField: "content_vector"
 namespace: "default"
}

Namespace Routing

When namespaces is configured, it is a map of namespace names to embedding field names. Each document is upserted into every configured namespace using the corresponding embedding field. This enables multi-vector indexing — for example, title embeddings in one namespace and body embeddings in another:

Weaviate Indexer

Mon, 09 Jun 2025 00:00:00 +0000

com.kmwllc.lucille.weaviate.indexer.WeaviateIndexer

Indexes documents into Weaviate. Config block: weaviate { ... }

Maven dependency:

<dependency>
 <groupId>com.kmwllc</groupId>
 <artifactId>lucille-weaviate</artifactId>
 <version>${lucille.version}</version>
</dependency>

Parameter	Type	Required	Description
`apiKey`	String	Yes	Weaviate API key for authentication. Use `${?WEAVIATE_API_KEY}` for environment variable substitution.
`host`	String	Yes	Weaviate instance hostname (e.g., `my-cluster.weaviate.network`). Do not include the protocol — HTTPS is used automatically.
`className`	String	No	The Weaviate class (object type) to create or update. Default: `"Document"`.
`idDestinationName`	String	No	Field name under which the document’s original Lucille ID is stored in Weaviate (since Weaviate’s `id` field must be a UUID). Default: `"id_original"`.
`vectorField`	String	No	Document field containing the vector embedding to index. If omitted, no vector is sent (useful if Weaviate is configured to generate its own embeddings).

indexer {
 class: "com.kmwllc.lucille.weaviate.indexer.WeaviateIndexer"
}

weaviate {
 apiKey: ${WEAVIATE_API_KEY}
 host: "my-cluster.weaviate.network"
 className: "Article"
 idDestinationName: "lucille_id"
 vectorField: "content_vector"
}

Note on IDs: Weaviate requires UUIDs for its internal id field. The WeaviateIndexer generates a UUID from the document’s Lucille ID and stores the original ID under idDestinationName so it can be retrieved later.