Indexer

Indexers

An indexer sends processed Documents to a specific destination. Only one Indexer can be defined in a Lucille run. All pipelines will feed to the same Indexer.

Indexer configuration has two parts: the generic indexer configuration, and configuration for the implementation you are using. For example, if you are using Solr, you’d provide solr config, or elastic for Elasticsearch, csv for CSV, etc.

Here’s what using the SolrIndexer might look like:

# Generic indexer config
indexer {
  type: "solr"
  ignoreFields: ["city_temp"]
  batchSize: 100
}
# Specific implementation (Solr) config
solr {
  useCloudClient: true
  url: "localhost:9200"
  defaultCollection: "test_index"
}

At a minimum, indexer must contain either type or class. type is shorthand for an indexer provided by lucille-core - it can be "Solr", "OpenSearch", "ElasticSearch", or "CSV". indexer can contain a variety of additional properties as well. Some Indexers do not support certain properties, however. For example, OpenSearchIndexer and ElasticsearchIndexer do not support indexer.indexOverrideField.

The lucille-core module contains a number of commonly used indexers. Additional indexers with a large number of dependencies are provided as optional plugin modules.

Lucille Indexers (Core)

• Solr Indexer
• OpenSearch Indexer
• Elasticsearch Indexer
• CSV Indexer

Lucille Indexers (Plugins)

• Pinecone Indexer
• Weaviate Indexer