Cookbooks

• 1: File Ingestion Cookbook
• 2: Vector Search Cookbook
• 3: RSS Cookbook

1 - File Ingestion Cookbook

This cookbook covers common file ingestion patterns using Lucille’s FileConnector.

Recipe 1: Ingest CSV Files from Local Filesystem

Read all .csv files in a directory and index each row as a document into OpenSearch.

connectors: [
  {
    name: "csv-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "csv-pipeline"
    paths: ["/data/csvfiles"]

    fileHandlers: {
      csv {
        filenameField: "source_file"
        docIdPrefix: "row-"
      }
    }

    filterOptions: {
      includes: [".*\\.csv$"]
    }
  }
]

pipelines: [
  {
    name: "csv-pipeline"
    stages: [
      {
        class: "com.kmwllc.lucille.stage.TrimWhitespace"
        fields: ["title", "description"]
      }
    ]
  }
]

indexer { type: "opensearch" }

opensearch {
  url: "https://localhost:9200"
  index: "my-docs"
  acceptInvalidCert: true
}

Recipe 2: Ingest JSON Files

Read .json or .jsonl files. Each top-level JSON object (or each line in a .jsonl file) becomes a Document.

connectors: [
  {
    name: "json-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "json-pipeline"
    paths: ["/data/jsonfiles"]

    fileHandlers: {
      json {}
    }
  }
]

For .jsonl files (one JSON object per line), no extra configuration is needed — the JSONFileHandler handles both formats automatically.

Recipe 3: Ingest XML Files

Extract records from XML files using an XPath expression.

connectors: [
  {
    name: "xml-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "xml-pipeline"
    paths: ["/data/xmlfiles"]

    fileHandlers: {
      xml {
        chunkPath: "//record"
      }
    }
  }
]

Each XML element matching //record becomes a separate Lucille Document.

Recipe 4: Ingest from Amazon S3

connectors: [
  {
    name: "s3-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "my-pipeline"
    paths: ["s3://my-bucket/data/"]

    s3 {
      accessKeyId: ${?AWS_ACCESS_KEY_ID}
      secretAccessKey: ${?AWS_SECRET_ACCESS_KEY}
      region: ${?AWS_DEFAULT_REGION}
    }

    fileHandlers: {
      csv {}
    }
  }
]

For paths with special characters (e.g., spaces), percent-encode the URI: s3://my-bucket/folder%20with%20spaces/.

Recipe 5: Ingest from Azure Blob Storage

connectors: [
  {
    name: "azure-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "my-pipeline"
    paths: ["https://mystorageaccount.blob.core.windows.net/my-container/"]

    azure {
      connectionString: ${?AZURE_CONNECTION_STRING}
    }
  }
]

Alternatively, authenticate with account name and key:

azure {
  accountName: ${?AZURE_ACCOUNT_NAME}
  accountKey: ${?AZURE_ACCOUNT_KEY}
}

Recipe 6: Ingest from Google Cloud Storage

connectors: [
  {
    name: "gcs-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "my-pipeline"
    paths: ["gs://my-bucket/data/"]

    gcp {
      pathToServiceKey: ${?GCP_SERVICE_KEY_PATH}
    }
  }
]

Recipe 7: Incremental Ingest (Only New or Modified Files)

Use state tracking to skip files that have not changed since the last run.

connectors: [
  {
    name: "incremental-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "my-pipeline"
    paths: ["/data/files"]
    publishMode: "INCREMENTAL"

    # JDBC state database (H2 embedded is the default; Derby also works)
    state {
      driver: "org.h2.Driver"
      connectionString: "jdbc:h2:./lucille-state"
    }

    filterOptions: {
      # Only process files that were last published more than 1 hour ago
      lastPublishedCutoff: "1h"
    }
  }
]

Notes on incremental mode:

• FULL mode (the default) republishes every file on every run.
• INCREMENTAL mode requires a state database and only processes files that are new or modified since the last run.
• The state database tracks when each file was last published by Lucille.

Recipe 8: Tombstone Deletions

Automatically detect deleted files and issue deletes against the search backend.

connectors: [
  {
    name: "connector-with-tombstones"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "my-pipeline"
    paths: ["/data/files"]
    publishMode: "INCREMENTAL"
    sendTombstones: true

    state {
      driver: "org.h2.Driver"
      connectionString: "jdbc:h2:./lucille-state"
    }
  }
]

indexer {
  type: "opensearch"
  deletionMarkerField: "file_expired"
  deletionMarkerFieldValue: "true"
}

When a file is deleted from the filesystem, Lucille creates a tombstone Document with file_expired: true. The OpenSearchIndexer sees this marker and issues a delete request against the search backend.

Recipe 9: Extract Text from PDF, Office, and Other Formats (Tika)

Use the lucille-tika plugin to extract text from arbitrary file formats. Set getFileContent: true in fileOptions to read raw bytes into file_content, then pass it to the TextExtractor stage.

connectors: [
  {
    name: "docs-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "docs-pipeline"
    paths: ["/data/documents"]

    fileOptions: {
      getFileContent: true
    }
  }
]

pipelines: [
  {
    name: "docs-pipeline"
    stages: [
      {
        class: "com.kmwllc.lucille.tika.stage.TextExtractor"
        source: "file_content"
        dest: "extracted_text"
      }
    ]
  }
]

Requires the lucille-tika Maven dependency. See TextExtractor in All Stages for setup.

Filter Options Reference

Use filterOptions to control which files are processed:

filterOptions: {
  # Only process files whose names match these patterns (regex)
  includes: [".*\\.pdf$", ".*\\.docx$"]

  # Skip files whose names match these patterns (regex)
  excludes: [".*\\.DS_Store$", ".*\\.tmp$"]

  # Only process files modified within the last 3 days
  lastModifiedCutoff: "3d"

  # Only process files not published by Lucille in the last 6 hours (requires state)
  lastPublishedCutoff: "6h"
}

Duration values accept: s (seconds), m (minutes), h (hours), d (days).

File Options Reference

fileOptions controls traversal behavior. fileHandlers (a separate block) configures which file types to process and how:

fileOptions: {
  # Read file bytes into the file_content field (slow; downloads cloud files)
  getFileContent: true

  # Process zip/tar archives by extracting their contents
  handleArchivedFiles: true

  # Process gzip-compressed files
  handleCompressedFiles: true

  # Move files to this path after successful processing (local files only)
  moveToAfterProcessing: "/data/processed"

  # Move files to this path if an error occurs (local files only)
  moveToErrorFolder: "/data/errors"
}

fileHandlers: {
  csv { ... }
  json { ... }
  xml { ... }
}

2 - Vector Search Cookbook

This cookbook shows how to build end-to-end vector search pipelines with Lucille.

Overview

A typical vector search pipeline:

1. Read source documents (files, database, RSS, etc.)
2. Extract text from each document if needed (Tika for PDF/Office files)
3. Chunk long text into smaller pieces suitable for embedding
4. Embed each chunk using an embedding model
5. Index the vectors into a vector database (Pinecone or Weaviate)

Recipe 1: CSV → OpenAI Embeddings → Pinecone

Index a CSV of articles into Pinecone using the OpenAI Embeddings API.

Prerequisites:

• lucille-pinecone plugin on the classpath
• An OpenAI API key in the OPENAI_API_KEY environment variable
• A Pinecone index configured to match the embedding model’s output dimensions

connectors: [
  {
    name: "article-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "embed-pipeline"
    paths: ["/data/articles.csv"]

    fileHandlers: {
      csv {
        docIdPrefix: "article-"
      }
    }
  }
]

pipelines: [
  {
    name: "embed-pipeline"
    stages: [
      # Combine title and body into a single text field for embedding
      {
        class: "com.kmwllc.lucille.stage.Concatenate"
        source: ["title", "body"]
        dest: "text_to_embed"
        delimiter: " "
      },
      # Generate a vector embedding using OpenAI
      {
        class: "com.kmwllc.lucille.stage.OpenAIEmbed"
        source: "text_to_embed"
        dest: "content_vector"
        modelName: "text-embedding-3-small"
        apiKey: ${OPENAI_API_KEY}
        embedDocument: true
        embedChildren: false
      }
    ]
  }
]

indexer {
  class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
}

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "articles-index"
  vectorField: "content_vector"
  namespace: "articles"
}

Recipe 2: PDF Files → Tika → Chunk → Local Embeddings (JLama) → Pinecone

Index PDF documents without sending data to an external API. The JlamaEmbed stage generates embeddings locally.

Prerequisites:

• lucille-tika plugin (for text extraction)
• lucille-jlama plugin (for local embedding)
• lucille-pinecone plugin (for indexing)

connectors: [
  {
    name: "pdf-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "pdf-embed-pipeline"
    paths: ["/data/pdfs"]

    filterOptions: {
      includes: [".*\\.pdf$"]
    }

    fileOptions: {
      getFileContent: true
    }
  }
]

pipelines: [
  {
    name: "pdf-embed-pipeline"
    stages: [
      # Extract text from PDF bytes using Apache Tika
      {
        class: "com.kmwllc.lucille.tika.stage.TextExtractor"
        source: "file_content"
        dest: "extracted_text"
      },
      # Split long text into smaller chunks (each chunk → child document)
      {
        class: "com.kmwllc.lucille.stage.ChunkText"
        source: "extracted_text"
        dest: "text"
        chunkingMethod: "sentence"
        chunksToMerge: 5
        chunksToOverlap: 1
        cleanChunks: true
        characterLimit: 2000
      },
      # Generate embeddings locally (no external API call) — only embed child chunks
      {
        class: "com.kmwllc.lucille.jlama.stage.JlamaEmbed"
        source: "text"
        dest: "content_vector"
        modelPath: "/models/all-MiniLM-L6-v2"
        embedDocument: false
        embedChildren: true
      }
    ]
  }
]

indexer {
  class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
}

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "pdf-index"
  vectorField: "content_vector"
}

Recipe 3: Parquet → Embeddings → Pinecone

Read pre-computed vector embeddings from a Parquet file (e.g., embeddings generated by an external model) and index them directly.

Prerequisites:

• lucille-parquet plugin
• lucille-pinecone plugin

connectors: [
  {
    name: "parquet-connector"
    class: "com.kmwllc.lucille.parquet.connector.ParquetConnector"
    pipeline: "vector-pipeline"
    path: "/data/embeddings.parquet"
    idField: "doc_id"
  }
]

pipelines: [
  {
    name: "vector-pipeline"
    stages: []
  }
]

indexer {
  class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
}

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "my-vectors"
  vectorField: "embedding"
}

Recipe 4: Chunking Strategy

The ChunkText Stage splits a long text field into chunks and emits each chunk as a child document. The child documents flow through all downstream stages and are indexed independently — the parent document is also indexed (unless dropped).

{
  class: "com.kmwllc.lucille.stage.ChunkText"
  source: "body"
  dest: "text"              # field name in child docs
  chunkingMethod: "sentence"
  chunksToMerge: 5          # merge 5 sentences into each final chunk
  chunksToOverlap: 1        # 1 sentence of overlap between adjacent chunks
  cleanChunks: true         # strip newlines and trim whitespace
  characterLimit: 2000      # hard cap per chunk
}

After chunking, each child document has:

• id: {parent-id}-chunk-{n} (e.g., doc-42-chunk-0, doc-42-chunk-1, …)
• parent_id: ID of the parent document
• chunk_number: zero-based index
• total_chunks: total number of chunks from this parent
• offset: character offset of this chunk in the original text
• length: character length of this chunk
• text (or whatever dest is set to): the chunk text

Parent document fields are not automatically copied to child documents.

Recipe 5: Conditional Embedding

Only generate embeddings for documents that have content, skipping short or empty documents:

{
  class: "com.kmwllc.lucille.stage.OpenAIEmbed"
  source: "content"
  dest: "content_vector"
  modelName: "text-embedding-3-small"
  apiKey: ${OPENAI_API_KEY}
  embedDocument: true
  embedChildren: false
  conditions: [
    { fields: ["content"], operator: "must" }
  ]
}

Recipe 6: Database → Ollama Summarization → Embeddings → OpenSearch

Enrich database records with LLM-generated summaries before embedding.

connectors: [
  {
    name: "db-connector"
    class: "com.kmwllc.lucille.connector.jdbc.DatabaseConnector"
    pipeline: "enrich-embed"
    driver: "org.postgresql.Driver"
    connectionString: "jdbc:postgresql://localhost:5432/mydb"
    sql: "SELECT id, title, body FROM documents"
    idField: "id"
  }
]

pipelines: [
  {
    name: "enrich-embed"
    stages: [
      # Generate a concise summary using a local Ollama model
      {
        class: "com.kmwllc.lucille.stage.PromptOllama"
        hostURL: "http://localhost:11434"
        modelName: "llama3"
        systemPrompt: "Summarize the following document in 2-3 sentences. Output only a JSON object with a single field: summary."
        fields: ["body"]
      },
      # Embed the summary
      {
        class: "com.kmwllc.lucille.stage.OpenAIEmbed"
        source: "summary"
        dest: "summary_vector"
        modelName: "text-embedding-3-small"
        apiKey: ${OPENAI_API_KEY}
        embedDocument: true
        embedChildren: false
      }
    ]
  }
]

indexer { type: "opensearch" }

opensearch {
  url: "https://localhost:9200"
  index: "enriched-docs"
  acceptInvalidCert: true
}

Model Comparison

Tips

• Batch size: Embedding API calls benefit from larger batches. The default indexer.batchSize (100) is a good starting point.
• Rate limiting: If you hit OpenAI rate limits, reduce worker.threads or add retry logic.
• Model dimensions: Your Pinecone/Weaviate index dimensions must match the embedding model’s output (e.g., text-embedding-3-small outputs 1536 dimensions).
• Testing: Use RandomVector Stage to generate fake embeddings during development without API calls.

3 - RSS Cookbook

RSS to CSV

Let’s say we wanted to read from an RSS feed into a CSV using Lucille. We can set this up in the following manner:

1. Create a .conf file to configure Lucille
2. Specify the RSS connector in the connectors section of the config:

connectors: [
  {
    name: "RSSConnector"
    pipeline: "rssPipeline"
    class: "com.kmwllc.lucille.connector.RSSConnector"
    rssURL: "https://www.cnbc.com/id/15837362/device/rss/rss.html"
  }
]

There are a few additional configuration options that we won’t use here, but are useful:

    useGuidForDocID: true       # defaults to true; set false to use UUID as ID instead
    pubDateCutoff: "24h"        # only publish items from the last 24 hours
    runDuration: "1h"           # run incrementally for 1 hour total
    refreshIncrement: "5m"      # re-fetch the feed every 5 minutes

Your pipeline name can be whatever you want. For our URL, we chose CNBC’s RSS feed.

3. Now we define what stages we would like to use to process our documents from the feed. To give context as to what these stages are doing:

News items in an RSS feed often have some article metadata, and then a link to the actual meat of the article in HTML as a field.

• The fetchURI stage allows us to grab the actual content of our associated news article.
• The ApplyJSoup stage parses that content into fields that will exist in addition to our article metadata from the RSS feed. These fields include the body, bullet points, and the header.

pipelines: [
  {
    name: "rssPipeline"
    stages: [
      {
        name: "fetchURI",
        class: "com.kmwllc.lucille.stage.FetchUri"
        source: "link"
        dest: "content"
      }
      {
        name: "ApplyJSoup"
        class: "com.kmwllc.lucille.stage.ApplyJSoup"
        byteArrayField: "content"
        destinationFields: {
          paragraphTexts: {
            type: "text",
            selector: ".ArticleBody-articleBody p"
          }
          bulletPoints: {
            type: "text",
            selector: ".RenderKeyPoints-list li"
          }
          headline: {
            type: "text"
            selector: "h1"
          }
        }
      }
    ]
  }
]

4. We can index these documents into whatever we’d like. Here, we might decide to just print them to a CSV:

indexer: {
  type: "csv"
}

csv: {
  path: "./rss_results.csv"
  columns: ["id", "link", "title", "description", "paragraphTexts",
    "bulletPoints", "headline"]
}

Here is the full config file:

connectors: [
  {
    name: "RSSConnector"
    pipeline: "rssPipeline"
    class: "com.kmwllc.lucille.connector.RSSConnector"
    rssURL: "https://www.cnbc.com/id/15837362/device/rss/rss.html"
  }
]

pipelines: [
  {
    name: "rssPipeline"
    stages: [
      {
        name: "fetchURI",
        class: "com.kmwllc.lucille.stage.FetchUri"
        source: "link"
        dest: "content"
      }
      {
        name: "ApplyJSoup"
        class: "com.kmwllc.lucille.stage.ApplyJSoup"
        byteArrayField: "content"
        destinationFields: {
          paragraphTexts: {
            type: "text",
            selector: ".ArticleBody-articleBody p"
          }
          bulletPoints: {
            type: "text",
            selector: ".RenderKeyPoints-list li"
          }
          headline: {
            type: "text"
            selector: "h1"
          }
        }
      }
    ]
  }
]

indexer: {
  type: "csv"
}

csv: {
  path: "./rss_results.csv"
  columns: ["id", "link", "title", "description", "paragraphTexts",
    "bulletPoints", "headline"]
}

The CSV file will thus be saved on disk.

RSS to OpenSearch

We might also choose to index into another destination, like an OpenSearch index. Here’s an example. Replace the config with this:

connectors: [
  {
    name: "RSSConnector"
    pipeline: "rssPipeline"
    class: "com.kmwllc.lucille.connector.RSSConnector"
    rssURL: "https://www.cnbc.com/id/15837362/device/rss/rss.html"
    refreshIncrement: "60s"
    runDuration: "1h"
  }
]

pipelines: [
  {
    name: "rssPipeline"
    stages: []
  }
]

indexer: {
  type: "opensearch"
}

opensearch: {
  url: <Your OpenSearch URL>
  index: "rss-index"
  acceptInvalidCert: true
}

You’ll notice we’re using incremental mode now, with a refreshIncrement of 60s and a runDuration of 1h. This means that every item in the feed will be indexed on the initial run, and then Lucille will continue to take in new items that pop up every 60 seconds for 1 hour total.

Run Lucille again. Here’s what 3 of our documents look like after being indexed into OpenSearch:

GET /rss-index/_search
{
  "size": 3
}

{
  "took": 15,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 30,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "rss-index",
        "_id": "108275704",
        "_score": 1,
        "_source": {
          "id": "108275704",
          "guid": "108275704",
          "isPermaLink": false,
          "link": "https://www.cnbc.com/2026/03/09/watch-live-trump-press-conference-iran-war-oil-hormuz-doral.html",
          "title": "Watch live: Trump holds press conference as Iran war fallout roils oil market",
          "pubDate": "2026-03-09T21:13:25Z",
          "run_id": "cb234bd2-fdf7-4b88-be13-20de36cd059e"
        }
      },
      {
        "_index": "rss-index",
        "_id": "108275619",
        "_score": 1,
        "_source": {
          "id": "108275619",
          "description": "The OpenAI deal fallout exposes the fundamental danger of being the most leveraged player.",
          "guid": "108275619",
          "isPermaLink": false,
          "link": "https://www.cnbc.com/2026/03/09/oracle-is-building-yesterdays-data-centers-with-tomorrows-debt.html",
          "title": "Oracle is building yesterday's data centers with tomorrow's debt",
          "pubDate": "2026-03-09T20:52:19Z",
          "run_id": "cb234bd2-fdf7-4b88-be13-20de36cd059e"
        }
      },
      {
        "_index": "rss-index",
        "_id": "108275649",
        "_score": 1,
        "_source": {
          "id": "108275649",
          "description": "U.S. stock market indexes rose on the heels of reported comments by President Donald Trump.",
          "guid": "108275649",
          "isPermaLink": false,
          "link": "https://www.cnbc.com/2026/03/09/trump-iran-war-end.html",
          "title": "Trump says Iran 'war is very complete,' talks to Putin, reports say",
          "pubDate": "2026-03-09T21:32:43Z",
          "run_id": "cb234bd2-fdf7-4b88-be13-20de36cd059e"
        }
      }
    ]
  }
}

Using other indexers with the RSS connector follows much the same pattern.