tensorflow::serving::LoaderHarness

#include <loader_harness.h>

LoaderHarness is a widget the Manager uses to hold on to and talk to a Loader while it owns it.

Summary

It tracks the overall state of a Servable such that Manager can determine which state transitions to make at what times.

A manager implementation can also add some additional state with each harness. For example, a manager could put ACL or lifecycle metadata here. The ownership is maintained by the harness.

This class is thread-safe.

Constructors and Destructors

LoaderHarness(const ServableId & id, std::unique_ptr< Loader > loader, const Options & options)
LoaderHarness(const ServableId & id, std::unique_ptr< Loader > loader, std::unique_ptr< T > additional_state, const Options & options)
Constructor to create a harness with additional state, which a manager needs.
~LoaderHarness()
Legal to destruct iff current state is one of kNew, kDisabled or kError.

Public types

State{
  kNew,
  kLoadRequested,
  kLoadApproved,
  kLoading,
  kReady,
  kUnloadRequested,
  kQuiescing,
  kQuiesced,
  kUnloading,
  kDisabled,
  kError
}
enum
State of the underlying servable, from the perspective of the LoaderHarness and for the purpose of communication between it and a Manager.

Public functions

DoneQuiescing()
Status
Transitions the state to kQuiesced, which means that there are no more live handles to this servable available in the frontend.
Error(const Status & status)
void
Transitions the state to kError and invokes 'options_.error_callback'.
Load()
Status
Transitions to kLoading, delegates to Servable::Load(), then transitions either to kReady if Load() succeeds, or to kError (and invokes 'options_.
LoadApproved()
Status
Transitions to kLoadApproved.
LoadRequested()
Status
Transitions the state of the harness to kLoadRequested iff its current state is kNew.
StartQuiescing()
Status
Transitions the state to kQuiescing, which means that we would like to not give out any more handles to this servable.
Unload()
Status
Transitions to kUnloading, delegates to Servable::Unload(), then transitions to kDisabled when Unload() is done.
UnloadRequested()
Status
Transitions the state of the harness to kUnloadRequested iff its current state is kReady.
additional_state()
T *
Gets the additional state.
cancel_load_retry()
bool
id() const
ServableId
Returns the identifier of underlying Servable.
loader() const
Loader *
Returns a pointer to the wrapped loader.
loader_state_snapshot()
ServableStateSnapshot< T >
Returns the current overall state snapshot of the underlying Servable.
set_cancel_load_retry(bool value)
void
Cancels retrying the load of the servable.
state()
Returns the current state of underlying Servable.
status()
Status
Whether anything has gone wrong with this servable.

Public static functions

StateDebugString(State state)
string

Structs

tensorflow::serving::LoaderHarness::Options

Options to configure a LoaderHarness.

Public types

State

 State

State of the underlying servable, from the perspective of the LoaderHarness and for the purpose of communication between it and a Manager.

Not equivalent to the semantic servable states in servable_state.h.

Valid transitions: kNew>kLoading>kReady>kQuiescing>kQuiesced>kUnloading>kDisabled as well as: any_state>kError.

Properties
kDisabled

'loader_->Unload()' has finished.

kError

An error has occurred, either during 'loader_->Load()' or outside of the harness (and was reported to the harness via a call to Error()).

kLoadApproved

The servable has been approved for loading, e.g.

resources have been set aside for it.

kLoadRequested

The manager has been requested to load this servable.

kLoading

'loader_->Load()' has been called.

kNew

Initial state.

kQuiesced

The servable has been made unavailable for serving.

kQuiescing

The servable is going to be made unavailable for serving.

kReady

'loader_->Load()' has succeeded.

kUnloadRequested

The manager has been requested to unload this servable.

kUnloading

'loader_->Unload()' has been called.

Public functions

DoneQuiescing

Status DoneQuiescing()

Transitions the state to kQuiesced, which means that there are no more live handles to this servable available in the frontend.

At this point, we can unload this object.

REQUIRES: State is kQuiescing when called. Otherwise DCHECK-fails, transitions to state kError and invokes 'options_.error_callback'.

Error

void Error(
  const Status & status
)

Transitions the state to kError and invokes 'options_.error_callback'.

Load

Status Load()

Transitions to kLoading, delegates to Servable::Load(), then transitions either to kReady if Load() succeeds, or to kError (and invokes 'options_.

error_callback') if Load() fails. This call may take a long time.

We retry the Servable::Load() according to the options set during construction of this class. We stop retrying and give up if 1. we have reached max_num_load_retries or, 2. if cancel_load() is set to true.

REQUIRES: State is kLoadApproved when called. Otherwise DCHECK-fails, transitions to state kError and invokes 'options_.error_callback'.

LoadApproved

Status LoadApproved()

Transitions to kLoadApproved.

REQUIRES: State is kLoadRequested when called. Otherwise DCHECK-fails, transitions to state kError and invokes 'options_.error_callback'.

LoadRequested

Status LoadRequested()

Transitions the state of the harness to kLoadRequested iff its current state is kNew.

The test-and-change is done transactionally, so this method can be used to ensure that at most one Load() request can proceed.

LoaderHarness

 LoaderHarness(
  const ServableId & id,
  std::unique_ptr< Loader > loader,
  const Options & options
)

LoaderHarness

 LoaderHarness(
  const ServableId & id,
  std::unique_ptr< Loader > loader,
  std::unique_ptr< T > additional_state,
  const Options & options
)

Constructor to create a harness with additional state, which a manager needs.

StartQuiescing

Status StartQuiescing()

Transitions the state to kQuiescing, which means that we would like to not give out any more handles to this servable.

REQUIRES: State is kUnloadRequested when called. Otherwise DCHECK-fails, transitions to state kError and invokes 'options_.error_callback'.

Unload

Status Unload()

Transitions to kUnloading, delegates to Servable::Unload(), then transitions to kDisabled when Unload() is done.

REQUIRES: State is kQuiesced when called. Otherwise DCHECK-fails, transitions to state kError and invokes 'options_.error_callback'.

UnloadRequested

Status UnloadRequested()

Transitions the state of the harness to kUnloadRequested iff its current state is kReady.

The test-and-change is done transactionally, so this method can be used to ensure that at most one Load() request can proceed.

additional_state

T * additional_state()

Gets the additional state.

Returns nullptr if the type mismatches or if it wasn't set.

cancel_load_retry

bool cancel_load_retry()

id

ServableId id() const 

Returns the identifier of underlying Servable.

loader

Loader * loader() const 

Returns a pointer to the wrapped loader.

Ownership remains with this class.

loader_state_snapshot

ServableStateSnapshot< T > loader_state_snapshot()

Returns the current overall state snapshot of the underlying Servable.

set_cancel_load_retry

void set_cancel_load_retry(
  bool value
)

Cancels retrying the load of the servable.

This is best-effort, and does not preempt a Load() which is already happening, only subsequent calls.

If the retries are cancelled, the servable goes into a state dependent on the last Load() called on it. If the last Load() was successful, it will be in state kReady, else in kError.

state

State state()

Returns the current state of underlying Servable.

status

Status status()

Whether anything has gone wrong with this servable.

If state is kError, this will be non-OK. If not OK, the error could be something that occurred in a Source or SourceAdapter, in the Loader, in the Manager, or elsewhere. All errors pertaining to the servable are reported here, regardless of origin.

~LoaderHarness

 ~LoaderHarness()

Legal to destruct iff current state is one of kNew, kDisabled or kError.

Check-fails if violated.

Public static functions

StateDebugString

string StateDebugString(
  State state
)