Publisher

Provides a way to publish Documents for processing by the pipeline, and tracks their lifecycle until completion.

The Publisher is the internal accounting system that tracks every Document from the moment it is submitted to the pipeline until it reaches a terminal state (indexed, failed, or dropped).

What the Publisher Does

Accepts documents from a Connector via publisher.publish(doc).
Stamps the run ID on each document before it enters the pipeline.
Registers the document in its accounting ledger so it can track completion.
Buffers documents on the source queue for Workers to consume.
Receives events (FINISH, FAIL, DROP, CREATE) from Workers and the Indexer.
Determines run completion when all submitted documents have reached a terminal state.

Run Completion

The Publisher declares a run complete only when all three of the following are simultaneously true:

The Connector thread has finished publishing all documents.
All document IDs in the accounting ledger have been accounted for (each has a terminal event).
The event queue is drained (no more events arriving).

This ensures that even out-of-order events and child documents generated mid-pipeline are correctly accounted for before the run is declared complete.

The internal accounting ledger is a Bag (multiset), not a Set. This means a Connector can legitimately publish two documents with the same ID in a single run — each is tracked independently and must individually reach a terminal event before the run completes.

Document Lifecycle Events

Event	Meaning
`CREATE`	A child document was generated by a Stage and needs to be tracked.
`FINISH`	A document was successfully indexed.
`FAIL`	A document failed during pipeline processing or indexing.
`DROP`	A document was explicitly dropped and will not be indexed.

Backpressure

The Publisher implements backpressure to prevent a fast Connector from overwhelming the system:

In local mode: publisher.queueCapacity bounds the in-memory source and destination queues. publish() blocks when the queue is full.
In distributed mode: publisher.maxPendingDocs blocks publish() when too many documents are in flight (pending completion).

publisher {
  # Local mode: max docs in each queue (source and destination queues share this limit)
  queueCapacity: 10000

  # Distributed mode: block the Connector when this many docs are pending
  maxPendingDocs: 80000
}

Collapsing Mode

When a Connector emits multiple consecutive Documents with the same user-visible ID (e.g., a CDC stream with multiple updates to the same record), the Publisher can merge them into a single Document with multi-valued fields before passing them to the pipeline. This is enabled by setting requiresCollapsingPublisher() to true in the Connector implementation.

numReceived counts every call to publisher.publish().
numPublished counts only the documents actually sent downstream after collapsing.

Run Statistics

The Publisher tracks the following counts for each run:

Stat	Description
`numPublished`	Documents submitted to the pipeline (after collapsing).
`numReceived`	Total calls to `publish()` (before collapsing).
`numPending`	Documents currently in flight (submitted but not yet terminal).
`numSucceeded`	Documents that reached the Indexer successfully.
`numFailed`	Documents that failed during processing or indexing.
`numDropped`	Documents explicitly dropped by a Stage.

These are reported in the run summary at completion:

connector1: complete. 200000 docs succeeded. 0 docs failed. 0 docs dropped.

Pause and Resume

The Publisher supports pausing and resuming document publication. publish() blocks when paused and wakes when resume() is called. This is used internally in some specialized deployment patterns.

Event Handling in Distributed Mode

In local mode, events flow through an in-memory queue. In distributed mode, events flow through a dedicated Kafka event topic. The topic name is derived from the run ID, ensuring isolation between concurrent runs.

See Events for more details.

Publisher Accounting

The Bag data structure, out-of-order event handling, the waitForCompletion loop, backpressure, and thread safety.

You are viewing documentation for an older version of Lucille.