Ingest Designer Guide

This guide is for anyone using Lucille to accomplish a search ingestion task. You’ll be writing configuration files that define one or more ingests — choosing connectors, composing pipelines, configuring indexers, and tuning run parameters. No Java code is required.

For conceptual explanations of how these components work and why they are designed the way they are, see Architecture.

Getting Started

• Writing a Config — Anatomy of a Lucille config file: required elements, available settings, validation, and HOCON basics.
• Defining Pipelines — Pipeline syntax, connecting connectors to pipelines, multiple pipelines, conditions, and stage reuse patterns.
• Control Flow — How to control what happens to a document as it moves through a pipeline: conditions, skipping, dropping, errors, child documents, and connector sequencing.

Component Reference

• Connectors — Common parameters, sequencing, and the full catalogue of built-in connectors (File, Database, Kafka, RSS, Solr, Parquet).
• Stages — Stage configuration, conditions reference, and the complete stage catalogue organized by category.
• Indexers — Generic indexer parameters, field filtering, deletion mechanics, and backend-specific configuration (Solr, OpenSearch, Elasticsearch, CSV, Pinecone, Weaviate).

Cookbooks

• File Ingestion — Ingest files from local, S3, Azure, or GCS. Covers CSV, JSON, XML, incremental mode, tombstones, and Tika text extraction.
• Vector Search — Build end-to-end vector search pipelines: chunk text, generate embeddings, and index into Pinecone or Weaviate.
• RSS Ingestion — Ingest RSS feeds into CSV or OpenSearch, including incremental mode.