Event

Thu, 23 Oct 2025 00:00:00 +0000

Lucille Events

As a Document passes through the Lucille pipeline, Event messages are generated at key transitions. The Publisher consumes these events to track the lifecycle of every document in the run and determine when all work is complete.

Event Types

Event	Who Sends It	Meaning
`CREATE`	Worker (on behalf of a Stage)	A child document was generated inside the pipeline and must be tracked.
`FINISH`	Indexer	A document was successfully sent to the search backend.
`FAIL`	Worker or Indexer	A document failed during processing or indexing.
`DROP`	Worker	A document was explicitly dropped and will not be indexed.

Events include the document’s id and run_id, allowing the Publisher to match each event to its corresponding accounting entry.

Worker

Thu, 23 Oct 2025 00:00:00 +0000

A Worker is a thread that pulls Documents from the source queue, runs them through a Pipeline of Stages, and pushes the processed results onto the destination queue for the Indexer to consume.

What the Worker Does

When a Worker starts, it:

Constructs its own instance of the configured Pipeline (including a private instance of every Stage).
Enters a polling loop, pulling Documents from the source queue one at a time.
Passes each Document through every Stage in the Pipeline in order.
Pushes the processed Document (and any child documents) to the destination queue.
Sends lifecycle events (FINISH, FAIL, DROP, CREATE) to the Publisher via the event queue.

Per-Thread Pipeline Isolation