Parquet Connector

Source Code

The ParquetConnector reads Apache Parquet files — locally or from Amazon S3 — and publishes each row as a Lucille Document. Parquet is a columnar format commonly used to store pre-computed embeddings, feature vectors, and large datasets.

Maven dependency:

<dependency>
  <groupId>com.kmwllc</groupId>
  <artifactId>lucille-parquet</artifactId>
  <version>${lucille.version}</version>
</dependency>

Configuration

connectors: [
  {
    name: "parquet-source"
    class: "com.kmwllc.lucille.parquet.connector.ParquetConnector"
    pipeline: "my-pipeline"
    pathToStorage: "/data/embeddings.parquet"
    idField: "doc_id"
    fsUri: "file:///"
  }
]

Configuration Parameters

S3 Configuration

For S3, provide the filesystem URI and credentials:

connectors: [
  {
    name: "parquet-s3"
    class: "com.kmwllc.lucille.parquet.connector.ParquetConnector"
    pipeline: "my-pipeline"
    pathToStorage: "/prefix/embeddings"
    idField: "doc_id"
    fsUri: "s3a://my-bucket"
    s3Key: ${AWS_ACCESS_KEY_ID}
    s3Secret: ${AWS_SECRET_ACCESS_KEY}
  }
]

Notes

• The connector uses Hadoop’s filesystem abstraction (FileSystem) for path traversal. Unlike FileConnector, it does not use Lucille’s StorageClient infrastructure.
• Parquet files must have the .parquet extension to be processed.
• When paginating with start/limit, it is recommended to use individual Connectors for each Parquet file rather than a directory path.
• The Parquet format requires random-access reads (not sequential streaming), which is why it is implemented as a standalone Connector rather than a FileConnector FileHandler.