tfx.orchestration.experimental.core.task_queue.TaskQueue

A thread-safe task queue with duplicate detection.

tfx.orchestration.experimental.core.task_queue.TaskQueue()

The life-cycle of a task starts with producers calling enqueue. Consumers call dequeue to obtain the tasks in FIFO order. When processing is complete, consumers must release the tasks by calling task_done.

Methods

`contains_task_id`

View source

contains_task_id(
    task_id: task_lib.TaskId
) -> bool

Returns True if the task queue contains a task with the given task_id.

Args
`task_id`	A task id.

Returns
`True` if a task with `task_id` was enqueued but `task_done` has not been invoked yet.

`dequeue`

View source

dequeue(
    max_wait_secs: Optional[float] = None
) -> Optional[tfx.orchestration.experimental.core.task.Task]

Removes and returns a task from the queue.

Once the processing is complete, queue consumers must call task_done.

Args
`max_wait_secs`	If not `None`, waits a maximum of `max_wait_secs` when the queue is empty for a task to be enqueued. If no task is present in the queue after the wait, `None` is returned. If `max_wait_secs` is `None` (default), returns `None` without waiting when the queue is empty.

Returns
A `Task` or `None` if the queue is empty.

`enqueue`

View source

enqueue(
    task: tfx.orchestration.experimental.core.task.Task
) -> bool

Enqueues the given task if no prior task with the same id exists.

Args
`task`	A `Task` object.

Returns
`True` if the task could be enqueued. `False` if a task with the same id already exists.

`is_empty`

View source

is_empty() -> bool

Returns True if the task queue is empty.

Queue is considered empty only if any enqueued tasks have been dequeued and task_done invoked on them.

`task_done`

View source

task_done(
    task: tfx.orchestration.experimental.core.task.Task
) -> None

Marks the processing of a task as done.

Consumers should call this method after the task is processed.

Args
`task`	A `Task` object.

Raises
`RuntimeError`	If attempt is made to mark a non-existent or non-dequeued task as done.