Connectors on Lucille

File Connector

Fri, 28 Feb 2025 00:00:00 +0000

The FileConnector traverses a file system and publishes a Lucille Document for each file it encounters. It supports local filesystems, Amazon S3, Azure Blob Storage, and Google Cloud Storage through a unified interface — a single connector config can traverse paths across multiple providers simultaneously. Optional File Handlers extract structured Documents from files that themselves contain data (CSV rows, JSON objects, XML elements).

Cloud Storage Configuration

When traversing cloud storage, provide authentication under the appropriate top-level config block alongside your connector config. Each provider also accepts an optional maxNumOfPages to limit how many file listings are loaded into memory per request.

Database Connector

Mon, 09 Jun 2025 00:00:00 +0000

Source Code

The DatabaseConnector reads rows from any JDBC-compatible relational database and publishes each row as a Lucille Document. Column names become field names on the Document.

Basic Configuration

connectors: [
 {
 name: "db-connector"
 class: "com.kmwllc.lucille.connector.jdbc.DatabaseConnector"
 pipeline: "my-pipeline"

 driver: "org.postgresql.Driver"
 connectionString: "jdbc:postgresql://localhost:5432/mydb"
 jdbcUser: "username"
 jdbcPassword: ${?DB_PASSWORD}
 sql: "SELECT id, title, body, published_at FROM articles WHERE active = true"
 idField: "id"
 }
]

Configuration Parameters

Parameter	Type	Required	Description
`driver`	String	Yes	JDBC driver class name. The driver JAR must be on the classpath.
`connectionString`	String	Yes	JDBC connection URL.
`jdbcUser`	String	No	Database username.
`jdbcPassword`	String	No	Database password. Use `${?VAR}` for environment variable substitution.
`sql`	String	Yes	SELECT statement to execute. All returned rows are published as Documents.
`idField`	String	No	Column whose value becomes the Document ID. If omitted, a UUID is generated per row.
`docIdPrefix`	String	No	Prefix prepended to every Document ID.
`fetchSize`	Integer	No	JDBC fetch size hint for streaming large result sets. For MySQL, set to `Integer.MIN_VALUE` (i.e., `-2147483648`) to avoid buffering the full result set in memory.
`preSQL`	String	No	A SQL statement (INSERT, DELETE, UPDATE, or DDL) executed once before the main query. Useful for creating temp tables, acquiring locks, or seeding data.
`postSQL`	String	No	A SQL statement executed once after the main query completes successfully. Useful for cleanup, releasing locks, or writing completion markers.
`otherSQLs`	List<String>	No	Additional SELECT queries to JOIN onto the primary result. Each query must return rows ordered by its join key.
`otherJoinFields`	List<String>	No	Join fields parallel to `otherSQLs`. Required when `otherSQLs` is specified. Must be integer-valued columns.
`ignoreColumns`	List<String>	No	Column names to skip when populating Documents.
`connectionRetries`	Integer	1	Number of connection retry attempts on failure.
`connectionRetryPause`	Integer	10000	Milliseconds to wait between connection retries.

Pre and Post SQL

Use preSQL and postSQL to run setup and teardown logic that must happen before and after the main query:

Solr Connector

Mon, 09 Jun 2025 00:00:00 +0000

Source Code

The SolrConnector issues a query against a Solr collection and publishes each result as a Lucille Document. It is useful for cross-index enrichment pipelines — for example, reading documents from one Solr collection, enriching them, and indexing them into a different collection or backend.

Internally, the connector uses cursor-based pagination (cursorMark) to iterate through large result sets without loading the full result into memory. Results are sorted by idField ascending (required for cursor pagination).

Kafka Connector

Mon, 09 Jun 2025 00:00:00 +0000

Source Code

The KafkaConnector reads Documents from a Kafka topic and publishes them into a Lucille pipeline. This is distinct from Kafka’s role as the messaging layer in distributed mode — the KafkaConnector is a data source, reading documents produced by an upstream system.

Use Cases

Streaming ingest from Kafka: An upstream application publishes documents (as JSON) to a Kafka topic, and Lucille reads them for enrichment and indexing.
Connectorless distributed mode: In this deployment pattern, a third-party publisher puts documents onto a Kafka source topic, and Lucille Workers consume them directly. In this case the KafkaConnector is not used — Workers listen to the source topic directly.

Configuration

All Kafka connection parameters are nested under the kafka key within the connector config block.

Parquet Connector

Mon, 09 Jun 2025 00:00:00 +0000

Source Code

The ParquetConnector reads Apache Parquet files — locally or from Amazon S3 — and publishes each row as a Lucille Document. Parquet is a columnar format commonly used to store pre-computed embeddings, feature vectors, and large datasets.

Maven dependency:

<dependency>
 <groupId>com.kmwllc</groupId>
 <artifactId>lucille-parquet</artifactId>
 <version>${lucille.version}</version>
</dependency>

Configuration

connectors: [
 {
 name: "parquet-source"
 class: "com.kmwllc.lucille.parquet.connector.ParquetConnector"
 pipeline: "my-pipeline"
 pathToStorage: "/data/embeddings.parquet"
 idField: "doc_id"
 fsUri: "file:///"
 }
]

Configuration Parameters

Parameter	Type	Required	Description
`pathToStorage`	String	Yes	Path to a Parquet file or directory to traverse for `.parquet` files.
`idField`	String	Yes	Field name in the Parquet schema to use as the Document ID. Must exist in the file’s schema.
`fsUri`	String	Yes	URI for the filesystem to use (e.g., `"file:///"` for local, `"s3a://my-bucket"` for S3).
`s3Key`	String	No	AWS S3 access key. Required when using S3.
`s3Secret`	String	No	AWS S3 secret key. Required when using S3.
`limit`	Long	No	Maximum number of Documents to publish. Default: no limit.
`start`	Long	No	Number of rows to skip from the beginning of each file. Default: `0`.

S3 Configuration

For S3, provide the filesystem URI and credentials:

RSS Connector

Fri, 06 Jun 2025 00:00:00 +0000

The RSSConnector

The RSSConnector allows you to publish Documents representing the items found in an RSS feed of your choice. Each Document will (optionally) contain fields from the RSS items, like the author, description, title, etc. By default, the Document IDs will be the item’s guid, which should be a unique identifier for the RSS item.

You can configure the RSSConnector to only publish recent RSS items, based on the pubDate found on the items. Also, it can run incrementally, refreshing the RSS feed after a certain amount of time until you manually stop it. The RSSConnector will avoid publishing Documents for the same RSS item more than once.