Vector Search Cookbook

This cookbook shows how to build end-to-end vector search pipelines with Lucille.

Overview

A typical vector search pipeline:

1. Read source documents (files, database, RSS, etc.)
2. Extract text from each document if needed (Tika for PDF/Office files)
3. Chunk long text into smaller pieces suitable for embedding
4. Embed each chunk using an embedding model
5. Index the vectors into a vector database (Pinecone or Weaviate)

Recipe 1: CSV → OpenAI Embeddings → Pinecone

Index a CSV of articles into Pinecone using the OpenAI Embeddings API.

Prerequisites:

• lucille-pinecone plugin on the classpath
• An OpenAI API key in the OPENAI_API_KEY environment variable
• A Pinecone index configured to match the embedding model’s output dimensions

connectors: [
  {
    name: "article-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "embed-pipeline"
    paths: ["/data/articles.csv"]

    fileHandlers: {
      csv {
        docIdPrefix: "article-"
      }
    }
  }
]

pipelines: [
  {
    name: "embed-pipeline"
    stages: [
      # Combine title and body into a single text field for embedding
      {
        class: "com.kmwllc.lucille.stage.Concatenate"
        source: ["title", "body"]
        dest: "text_to_embed"
        delimiter: " "
      },
      # Generate a vector embedding using OpenAI
      {
        class: "com.kmwllc.lucille.stage.OpenAIEmbed"
        source: "text_to_embed"
        dest: "content_vector"
        modelName: "text-embedding-3-small"
        apiKey: ${OPENAI_API_KEY}
        embedDocument: true
        embedChildren: false
      }
    ]
  }
]

indexer {
  class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
}

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "articles-index"
  vectorField: "content_vector"
  namespace: "articles"
}

Recipe 2: PDF Files → Tika → Chunk → Local Embeddings (JLama) → Pinecone

Index PDF documents without sending data to an external API. The JlamaEmbed stage generates embeddings locally.

Prerequisites:

• lucille-tika plugin (for text extraction)
• lucille-jlama plugin (for local embedding)
• lucille-pinecone plugin (for indexing)

connectors: [
  {
    name: "pdf-connector"
    class: "com.kmwllc.lucille.connector.FileConnector"
    pipeline: "pdf-embed-pipeline"
    paths: ["/data/pdfs"]

    filterOptions: {
      includes: [".*\\.pdf$"]
    }

    fileOptions: {
      getFileContent: true
    }
  }
]

pipelines: [
  {
    name: "pdf-embed-pipeline"
    stages: [
      # Extract text from PDF bytes using Apache Tika
      {
        class: "com.kmwllc.lucille.tika.stage.TextExtractor"
        source: "file_content"
        dest: "extracted_text"
      },
      # Split long text into smaller chunks (each chunk → child document)
      {
        class: "com.kmwllc.lucille.stage.ChunkText"
        source: "extracted_text"
        dest: "text"
        chunkingMethod: "sentence"
        chunksToMerge: 5
        chunksToOverlap: 1
        cleanChunks: true
        characterLimit: 2000
      },
      # Generate embeddings locally (no external API call) — only embed child chunks
      {
        class: "com.kmwllc.lucille.jlama.stage.JlamaEmbed"
        source: "text"
        dest: "content_vector"
        modelPath: "/models/all-MiniLM-L6-v2"
        embedDocument: false
        embedChildren: true
      }
    ]
  }
]

indexer {
  class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
}

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "pdf-index"
  vectorField: "content_vector"
}

Recipe 3: Parquet → Embeddings → Pinecone

Read pre-computed vector embeddings from a Parquet file (e.g., embeddings generated by an external model) and index them directly.

Prerequisites:

• lucille-parquet plugin
• lucille-pinecone plugin

connectors: [
  {
    name: "parquet-connector"
    class: "com.kmwllc.lucille.parquet.connector.ParquetConnector"
    pipeline: "vector-pipeline"
    path: "/data/embeddings.parquet"
    idField: "doc_id"
  }
]

pipelines: [
  {
    name: "vector-pipeline"
    stages: []
  }
]

indexer {
  class: "com.kmwllc.lucille.pinecone.indexer.PineconeIndexer"
}

pinecone {
  apiKey: ${PINECONE_API_KEY}
  index: "my-vectors"
  vectorField: "embedding"
}

Recipe 4: Chunking Strategy

The ChunkText Stage splits a long text field into chunks and emits each chunk as a child document. The child documents flow through all downstream stages and are indexed independently — the parent document is also indexed (unless dropped).

{
  class: "com.kmwllc.lucille.stage.ChunkText"
  source: "body"
  dest: "text"              # field name in child docs
  chunkingMethod: "sentence"
  chunksToMerge: 5          # merge 5 sentences into each final chunk
  chunksToOverlap: 1        # 1 sentence of overlap between adjacent chunks
  cleanChunks: true         # strip newlines and trim whitespace
  characterLimit: 2000      # hard cap per chunk
}

After chunking, each child document has:

• id: {parent-id}-chunk-{n} (e.g., doc-42-chunk-0, doc-42-chunk-1, …)
• parent_id: ID of the parent document
• chunk_number: zero-based index
• total_chunks: total number of chunks from this parent
• offset: character offset of this chunk in the original text
• length: character length of this chunk
• text (or whatever dest is set to): the chunk text

Parent document fields are not automatically copied to child documents.

Recipe 5: Conditional Embedding

Only generate embeddings for documents that have content, skipping short or empty documents:

{
  class: "com.kmwllc.lucille.stage.OpenAIEmbed"
  source: "content"
  dest: "content_vector"
  modelName: "text-embedding-3-small"
  apiKey: ${OPENAI_API_KEY}
  embedDocument: true
  embedChildren: false
  conditions: [
    { fields: ["content"], operator: "must" }
  ]
}

Recipe 6: Database → Ollama Summarization → Embeddings → OpenSearch

Enrich database records with LLM-generated summaries before embedding.

connectors: [
  {
    name: "db-connector"
    class: "com.kmwllc.lucille.connector.jdbc.DatabaseConnector"
    pipeline: "enrich-embed"
    driver: "org.postgresql.Driver"
    connectionString: "jdbc:postgresql://localhost:5432/mydb"
    sql: "SELECT id, title, body FROM documents"
    idField: "id"
  }
]

pipelines: [
  {
    name: "enrich-embed"
    stages: [
      # Generate a concise summary using a local Ollama model
      {
        class: "com.kmwllc.lucille.stage.PromptOllama"
        hostURL: "http://localhost:11434"
        modelName: "llama3"
        systemPrompt: "Summarize the following document in 2-3 sentences. Output only a JSON object with a single field: summary."
        fields: ["body"]
      },
      # Embed the summary
      {
        class: "com.kmwllc.lucille.stage.OpenAIEmbed"
        source: "summary"
        dest: "summary_vector"
        modelName: "text-embedding-3-small"
        apiKey: ${OPENAI_API_KEY}
        embedDocument: true
        embedChildren: false
      }
    ]
  }
]

indexer { type: "opensearch" }

opensearch {
  url: "https://localhost:9200"
  index: "enriched-docs"
  acceptInvalidCert: true
}

Model Comparison

Tips

• Batch size: Embedding API calls benefit from larger batches. The default indexer.batchSize (100) is a good starting point.
• Rate limiting: If you hit OpenAI rate limits, reduce worker.threads or add retry logic.
• Model dimensions: Your Pinecone/Weaviate index dimensions must match the embedding model’s output (e.g., text-embedding-3-small outputs 1536 dimensions).
• Testing: Use RandomVector Stage to generate fake embeddings during development without API calls.