tensorflow::serving::ServerCore

#include <server_core.h>

ServerCore contains state and helper methods enabling the building of ModelServers that support multiple interfaces.

Summary

All functionality in ServerCore is independent of any domain specific APIs and independent of platforms.

In terms of state, ServerCore is initialized with and retains a static ModelServerConfig, from which it bootstraps an AspiredVersionsManager and auxiliary data structures to support efficient serving.

Interfaces built above ServerCore, e.g. RPC service implementations, will remain stateless and will perform all lookups of servables (models) via ServerCore.

Inheritance

Inherits from: tensorflow::serving::Manager

Constructors and Destructors
~ServerCore()

Public types
CustomModelConfigLoader	using std::function< Status(const ::google::protobuf::Any &any, EventBus< ServableState > *event_bus, UniquePtrWithDeps< AspiredVersionsManager > *manager)> A function that's responsible for instantiating and connecting the necessary custom sources and source adapters to the manager based on a passed in config (any).
PreLoadHook	using AspiredVersionsManager::PreLoadHook
ServableStateMonitorCreator	using std::function< Status(EventBus< ServableState > *event_bus, std::unique_ptr< ServableStateMonitor > *monitor)>
ServerRequestLoggerUpdater	using std::function< Status(const ModelServerConfig &, ServerRequestLogger *)> Function signature used to update the server_request_logger.

Friend classes
test_util::ServerCoreTestAccess	friend class

Public functions
GetAvailableServableHandles() const	std::map< ServableId, ServableHandle< T > >
GetServableHandle(const ModelSpec & model_spec, ServableHandle< T > *const handle)	Status Returns a ServableHandle given a ModelSpec.
GetServableHandle(const ModelSpec & model_spec, ServableHandle< Servable > *const handle)	virtual Status
ListAvailableServableIds() const override	virtual std::vector< ServableId > Gets a list of all available servable ids, i.e.
Log(const google::protobuf::Message & request, const google::protobuf::Message & response, const LogMetadata & log_metadata)	virtual Status Writes the log for the particular request, response and metadata, if we decide to sample it and if request-logging was configured for the particular model.
ReloadConfig(const ModelServerConfig & config)	virtual Status Updates the server core with all the models and sources per the ModelServerConfig.
StartLoggingStream(const LogMetadata & log_metadata, ServerRequestLogger::CreateStreamLoggerFn< Request, Response > create_stream_logger_fn)	std::unique_ptr< StreamLogger< Request, Response > >
enable_cors_support() const	bool
predict_response_tensor_serialization_option() const	internal::PredictResponseTensorSerializationOption
servable_state_monitor() const	virtual ServableStateMonitor * Returns ServableStateMonitor that can be used to query servable states.

Public static functions
Create(Options options, std::unique_ptr< ServerCore > *core)	Status Creates a ServerCore instance with all the models and sources per the ModelServerConfig.

Protected functions
ServerCore(Options options)

Structs
tensorflow::serving::ServerCore::Options	Options for configuring a ServerCore object.

Public types

CustomModelConfigLoader

std::function< Status(const ::google::protobuf::Any &any, EventBus< ServableState > *event_bus, UniquePtrWithDeps< AspiredVersionsManager > *manager)> CustomModelConfigLoader

A function that's responsible for instantiating and connecting the necessary custom sources and source adapters to the manager based on a passed in config (any).

The expected pattern is that ownership of the created sources/source adapters can be transferred to the manager.

PreLoadHook

AspiredVersionsManager::PreLoadHook PreLoadHook

ServableStateMonitorCreator

std::function< Status(EventBus< ServableState > *event_bus, std::unique_ptr< ServableStateMonitor > *monitor)> ServableStateMonitorCreator

ServerRequestLoggerUpdater

std::function< Status(const ModelServerConfig &, ServerRequestLogger *)> ServerRequestLoggerUpdater

Function signature used to update the server_request_logger.

Friend classes

test_util::ServerCoreTestAccess

friend class test_util::ServerCoreTestAccess

Public functions

GetAvailableServableHandles

std::map< ServableId, ServableHandle< T > > GetAvailableServableHandles() const 

GetServableHandle

Status GetServableHandle(
  const ModelSpec & model_spec,
  ServableHandle< T > *const handle
)

Returns a ServableHandle given a ModelSpec.

Returns error if no such Servable is available e.g. not yet loaded, has been quiesced/unloaded, etc. Callers may assume that an OK status indicates a non-null handle.

IMPORTANT: The caller should only hold on to a handle for a short time, for example for the duration of a single request. Holding a handle for a long period of time will prevent servable loading and unloading.

If 'options_.allow_version_labels==true', recognizes two specific model version labels "stable" and "canary" and resolves them to the smallest and largest available version, respectively.

GetServableHandle

virtual Status GetServableHandle(
  const ModelSpec & model_spec,
  ServableHandle< Servable > *const handle
)

ListAvailableServableIds

virtual std::vector< ServableId > ListAvailableServableIds() const override

Gets a list of all available servable ids, i.e.

each of these can be retrieved using GetServableHandle.

Log

virtual Status Log(
  const google::protobuf::Message & request,
  const google::protobuf::Message & response,
  const LogMetadata & log_metadata
)

Writes the log for the particular request, response and metadata, if we decide to sample it and if request-logging was configured for the particular model.

ReloadConfig

virtual Status ReloadConfig(
  const ModelServerConfig & config
)

Updates the server core with all the models and sources per the ModelServerConfig.

Like Create(), waits for all statically configured servables to be made available before returning, and returns an error if any such model fails to load. (Does not necessarily wait for models removed from the config to finish unloading; that may occur asynchronously.)

IMPORTANT: It is only legal to call this method more than once if using ModelConfigList (versus custom model config).

StartLoggingStream

std::unique_ptr< StreamLogger< Request, Response > > StartLoggingStream(
  const LogMetadata & log_metadata,
  ServerRequestLogger::CreateStreamLoggerFn< Request, Response > create_stream_logger_fn
)

enable_cors_support

bool enable_cors_support() const 

predict_response_tensor_serialization_option

internal::PredictResponseTensorSerializationOption predict_response_tensor_serialization_option() const 

servable_state_monitor

virtual ServableStateMonitor * servable_state_monitor() const 

Returns ServableStateMonitor that can be used to query servable states.

~ServerCore

virtual  ~ServerCore()=default

Public static functions

Create

Status Create(
  Options options,
  std::unique_ptr< ServerCore > *core
)

Creates a ServerCore instance with all the models and sources per the ModelServerConfig.

For models statically configured with ModelConfigList, waits for them to be made available (or hit an error) for serving before returning. Returns an error status if any such model fails to load.

Protected functions

ServerCore

 ServerCore(
  Options options
)