Indexers

For conceptual documentation — what an Indexer is, why batching matters, deletion as a design pattern, and error handling at the batch level — see Architecture: Indexer.

Generic indexer Configuration

Indexer configuration has two parts: the generic indexer block (common to all backends), and a backend-specific config block (e.g., solr, opensearch, elastic, csv).

indexer {
  type: "Solr"
  batchSize: 100
  batchTimeout: 100
  blacklist: ["internal_field"]
}

solr {
  url: "http://localhost:8983/solr/my-collection"
}

indexer.type is shorthand for a built-in indexer: "Solr", "OpenSearch", "Elasticsearch", or "CSV". For plugin indexers, use indexer.class with the fully qualified class name instead.

Generic Parameters

Field Filtering

Whitelist and blacklist are applied at indexing time, not during pipeline processing. Stages see all fields on a document; only the Indexer strips fields before sending to the backend. Reserved internal fields (___dropped, ___skipped, ___children) are always stripped.

When indexOverrideField is set, the Indexer uses a MultiBatch — maintaining a separate batch per distinct field value and flushing each independently when it reaches batchSize or batchTimeout.

Deletion Mechanics

Two distinct deletion mechanisms are available:

Delete by ID: Set deletionMarkerField and deletionMarkerFieldValue. When a document has the marker field set to the marker value, the Indexer issues a delete-by-ID against the search backend for that document’s ID.

Delete by query: Also set deleteByFieldField and deleteByFieldValue. When a document has all four deletion fields set, the Indexer issues a delete-by-query: it deletes all documents in the index where deleteByFieldField’s referenced field equals the value in deleteByFieldValue’s referenced field.

indexer {
  type: "Solr"
  deletionMarkerField: "file_expired"
  deletionMarkerFieldValue: "true"
  deleteByFieldField: "delete_by_field"
  deleteByFieldValue: "delete_by_value"
}

Batching Behavior

Documents accumulate in a batch and are flushed when any of these conditions is met:

• The batch reaches batchSize documents (default: 100).
• The batch reaches batchByteSize bytes of accumulated document payload (disabled by default).
• batchTimeout milliseconds have elapsed since the last document was added or the last flush (default: 100ms).

When only batchByteSize is set (without batchSize), document-count batching is effectively disabled — batches flush purely on payload size. When both are set, whichever limit is reached first triggers the flush. This is useful for backends with request-size limits (e.g., a 10 MB bulk API limit) where a fixed document count may produce unpredictably sized payloads.

The timeout flush ensures documents are not left waiting indefinitely in low-volume scenarios.

Batch-level vs. per-document failures: A bulk-API failure that rejects the entire request fails all documents in the batch. Individual document rejections in the response (e.g., mapping errors) fail only those specific documents — the rest succeed. Both cases are tracked separately in the run summary.

Indexer Catalogue

Core Indexers

• Solr Indexer — Single-node and SolrCloud.
• OpenSearch Indexer — OpenSearch with optional partial updates.
• Elasticsearch Indexer — Elasticsearch with join field support.
• CSV Indexer — Write pipeline output to a CSV file.
• NopIndexer — No-op indexer for testing.

Plugin Indexers

• Pinecone Indexer — Index vector embeddings into Pinecone.
• Weaviate Indexer — Index documents and vectors into Weaviate.