You are viewing a snapshot of the in-development Lucille documentation.

This pre-release version reflects the current state of main and may contain unreleased changes.
For the stable release, see the latest version.

Parquet Connector

A Connector that reads Apache Parquet files and publishes each row as a Lucille Document.

Source Code

The ParquetConnector reads Apache Parquet files — locally or from Amazon S3 — and publishes each row as a Lucille Document. Parquet is a columnar format commonly used to store pre-computed embeddings, feature vectors, and large datasets.

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-parquet</artifactId>
  <version>${lucille.version}</version>
</dependency>

Configuration

connectors: [
  {
    name: "parquet-source"
    class: "com.kmwllc.lucille.parquet.connector.ParquetConnector"
    pipeline: "my-pipeline"
    pathToStorage: "/data/embeddings.parquet"
    idField: "doc_id"
    fsUri: "file:///"
  }
]

Configuration Parameters

ParameterTypeRequiredDescription
pathToStorageStringYesPath to a Parquet file or directory to traverse for .parquet files.
idFieldStringYesField name in the Parquet schema to use as the Document ID. Must exist in the file’s schema.
fsUriStringYesURI for the filesystem to use (e.g., "file:///" for local, "s3a://my-bucket" for S3).
s3KeyStringNoAWS S3 access key. Required when using S3.
s3SecretStringNoAWS S3 secret key. Required when using S3.
limitLongNoMaximum number of Documents to publish. Default: no limit.
startLongNoNumber of rows to skip from the beginning of each file. Default: 0.

S3 Configuration

For S3, provide the filesystem URI and credentials:

connectors: [
  {
    name: "parquet-s3"
    class: "com.kmwllc.lucille.parquet.connector.ParquetConnector"
    pipeline: "my-pipeline"
    pathToStorage: "/prefix/embeddings"
    idField: "doc_id"
    fsUri: "s3a://my-bucket"
    s3Key: ${AWS_ACCESS_KEY_ID}
    s3Secret: ${AWS_SECRET_ACCESS_KEY}
  }
]

Notes

  • The connector uses Hadoop’s filesystem abstraction (FileSystem) for path traversal. Unlike FileConnector, it does not use Lucille’s StorageClient infrastructure.
  • Parquet files must have the .parquet extension to be processed.
  • When paginating with start/limit, it is recommended to use individual Connectors for each Parquet file rather than a directory path.
  • The Parquet format requires random-access reads (not sequential streaming), which is why it is implemented as a standalone Connector rather than a FileConnector FileHandler.