Cookbooks on Lucille

File Ingestion Cookbook

Mon, 09 Jun 2025 00:00:00 +0000

This cookbook covers common file ingestion patterns using Lucille’s FileConnector.

Recipe 1: Ingest CSV Files from Local Filesystem

Read all .csv files in a directory and index each row as a document into OpenSearch.

connectors: [
 {
 name: "csv-connector"
 class: "com.kmwllc.lucille.connector.FileConnector"
 pipeline: "csv-pipeline"
 paths: ["/data/csvfiles"]

 fileHandlers: {
 csv {
 filenameField: "source_file"
 docIdPrefix: "row-"
 }
 }

 filterOptions: {
 includes: [".*\\.csv$"]
 }
 }
]

pipelines: [
 {
 name: "csv-pipeline"
 stages: [
 {
 class: "com.kmwllc.lucille.stage.TrimWhitespace"
 fields: ["title", "description"]
 }
 ]
 }
]

indexer { type: "opensearch" }

opensearch {
 url: "https://localhost:9200"
 index: "my-docs"
 acceptInvalidCert: true
}

Recipe 2: Ingest JSON Files

Read .json or .jsonl files. Each top-level JSON object (or each line in a .jsonl file) becomes a Document.

Vector Search Cookbook

Mon, 09 Jun 2025 00:00:00 +0000

This cookbook shows how to build end-to-end vector search pipelines with Lucille.

Overview

A typical vector search pipeline:

Read source documents (files, database, RSS, etc.)
Extract text from each document if needed (Tika for PDF/Office files)
Chunk long text into smaller pieces suitable for embedding
Embed each chunk using an embedding model
Index the vectors into a vector database (Pinecone or Weaviate)

Recipe 1: CSV → OpenAI Embeddings → Pinecone

Index a CSV of articles into Pinecone using the OpenAI Embeddings API.

RSS Cookbook

Tue, 10 Mar 2026 00:00:00 +0000

RSS to CSV

Let’s say we wanted to read from an RSS feed into a CSV using Lucille. We can set this up in the following manner:

Create a .conf file to configure Lucille
Specify the RSS connector in the connectors section of the config:

connectors: [
 {
 name: "RSSConnector"
 pipeline: "rssPipeline"
 class: "com.kmwllc.lucille.connector.RSSConnector"
 rssURL: "https://www.cnbc.com/id/15837362/device/rss/rss.html"
 }
]

There are a few additional configuration options that we won’t use here, but are useful: