Event

As Lucille runs, it generates Events that track document lifecycle and enable run completion accounting.

Lucille Events

As a Document passes through the Lucille pipeline, Event messages are generated at key transitions. The Publisher consumes these events to track the lifecycle of every document in the run and determine when all work is complete.

Event Types

Event	Who Sends It	Meaning
`CREATE`	Worker (on behalf of a Stage)	A child document was generated inside the pipeline and must be tracked.
`FINISH`	Indexer	A document was successfully sent to the search backend.
`FAIL`	Worker or Indexer	A document failed during processing or indexing.
`DROP`	Worker	A document was explicitly dropped and will not be indexed.

Events include the document’s id and run_id, allowing the Publisher to match each event to its corresponding accounting entry.

Event Flow

Worker → [event queue] → Publisher
Indexer → [event queue] → Publisher

In local mode, events flow through an in-memory queue on the main thread’s polling loop.

In distributed mode, events flow through a dedicated Kafka event topic. The topic name is derived from the run ID, ensuring that events from different concurrent runs are always isolated.

Event Topics (Kafka)

In distributed mode, each run creates its own event topic named based on the run ID and pipeline name. This ensures that the Publisher for a given run only sees events for its own documents. Because Workers and Indexers are long-running processes that serve multiple runs over their lifetime, a document’s run_id is the mechanism that routes its events to the correct Publisher — enabling multiple concurrent Runner invocations to share the same Worker and Indexer pool without their accounting interfering with each other. See Long-Running Workers and Indexers for the full operational pattern.

When kafka.events is set to false in the config, event messages are not sent to Kafka. This is appropriate only in streaming mode (no Runner) where run completion tracking is not needed.

kafka {
  events: true  # default; set to false only in pure streaming mode
}

Connector-less (Streaming) Mode

In connector-less distributed mode, a third-party publisher writes Documents directly to a Kafka source topic. There is no Lucille Runner or Publisher. In this case:

If kafka.events is true, the third-party publisher must include a run_id on each document (it can choose its own run ID value).
Workers and the Indexer send events to the Kafka event topic as usual.
Since there is no Publisher polling the event topic, events accumulate in the topic and are not consumed (unless you route them to your own consumer).

If run tracking is not needed in streaming mode, set kafka.events: false to suppress event production entirely.

Child Documents

Child documents generated by Stages must be registered with the Publisher before the parent document reaches the Indexer. The Worker sends a CREATE event for each child as soon as it is emitted, before the parent’s pipeline execution completes. This ordering guarantee ensures the Publisher never declares a run complete while child documents are still in flight.

Out-of-Order Events

The Publisher handles out-of-order events correctly. A child document can complete (receive a FINISH event from the Indexer) before the Publisher has even received the child’s CREATE event (because Workers and Indexers run concurrently). In this case, the Publisher stores the premature FINISH in a secondary buffer and reconciles it when the CREATE event subsequently arrives.

You are viewing a snapshot of the in-development Lucille documentation.