View source on GitHub
|
Specifies how to shard data among tf.data service workers.
tf.data.experimental.service.ShardingPolicy(
*args, **kwds
)
OFF: No sharding will be performed. Each worker produces the entire dataset without any sharding. With this mode, the best practice is to shuffle the dataset nondeterministically so that workers process the dataset in different orders. If workers are restarted or join the cluster mid-job, they will begin processing the dataset from the beginning.
DYNAMIC: The input dataset is dynamically split among workers at runtime. Each worker gets the next split when it reads data from the dispatcher. Data is produced non-deterministically in this mode. Dynamic sharding works well with varying-sized tf.data service clusters, e.g., when you need to auto-scale your workers. Dynamic sharding provides at-most once visitation guarantees. No examples will be repeated, but some may be missed if a tf.data service worker gets restarted while processing a file.
The following are static sharding policies. The semantics are similar to
tf.data.experimental.AutoShardPolicy. These policies require:
- The tf.data service cluster is configured with a fixed list of workers in DispatcherConfig.
- Each client only reads from the local tf.data service worker.
If a worker is restarted while performing static sharding, the worker will begin processing its shard again from the beginning.
FILE: Shards by input files (i.e. each worker will get a fixed set of files to process). When this option is selected, make sure that there is at least as many files as workers. If there are fewer input files than workers, a runtime error will be raised.
DATA: Shards by elements produced by the dataset. Each worker will process the whole dataset and discard the portion that is not for itself. Note that for this mode to correctly partition the dataset elements, the dataset needs to produce elements in a deterministic order.
FILE_OR_DATA: Attempts FILE-based sharding, falling back to DATA-based sharding on failure.
HINT: Looks for the presence of shard(SHARD_HINT, ...) which is treated as a
placeholder to replace with shard(num_workers, worker_index).
Methods
as_integer_ratio
as_integer_ratio()
Return integer ratio.
Return a pair of integers, whose ratio is exactly equal to the original int and with a positive denominator.
(10).as_integer_ratio()(10, 1)(-10).as_integer_ratio()(-10, 1)(0).as_integer_ratio()(0, 1)
bit_count
bit_count()
Number of ones in the binary representation of the absolute value of self.
Also known as the population count.
bin(13)'0b1101'(13).bit_count()3
bit_length
bit_length()
Number of bits necessary to represent self in binary.
bin(37)'0b100101'(37).bit_length()6
conjugate
conjugate()
Returns self, the complex conjugate of any int.
from_bytes
from_bytes(
byteorder='big', *, signed=False
)
Return the integer represented by the given array of bytes.
bytes Holds the array of bytes to convert. The argument must either support the buffer protocol or be an iterable object producing bytes. Bytes and bytearray are examples of built-in objects that support the buffer protocol. byteorder The byte order used to represent the integer. If byteorder is 'big', the most significant byte is at the beginning of the byte array. If byteorder is 'little', the most significant byte is at the end of the byte array. To request the native byte order of the host system, use `sys.byteorder' as the byte order value. Default is to use 'big'. signed Indicates whether two's complement is used to represent the integer.
to_bytes
to_bytes(
length=1, byteorder='big', *, signed=False
)
Return an array of bytes representing an integer.
length Length of bytes object to use. An OverflowError is raised if the integer is not representable with the given number of bytes. Default is length 1. byteorder The byte order used to represent the integer. If byteorder is 'big', the most significant byte is at the beginning of the byte array. If byteorder is 'little', the most significant byte is at the end of the byte array. To request the native byte order of the host system, use `sys.byteorder' as the byte order value. Default is to use 'big'. signed Determines whether two's complement is used to represent the integer. If signed is False and a negative integer is given, an OverflowError is raised.
__abs__
__abs__()
abs(self)
__add__
__add__(
value, /
)
Return self+value.
__and__
__and__(
value, /
)
Return self&value.
__bool__
__bool__()
True if self else False
__eq__
__eq__(
value, /
)
Return self==value.
__floordiv__
__floordiv__(
value, /
)
Return self//value.
__ge__
__ge__(
value, /
)
Return self>=value.
__gt__
__gt__(
value, /
)
Return self>value.
__invert__
__invert__()
~self
__le__
__le__(
value, /
)
Return self<=value.
__lshift__
__lshift__(
value, /
)
Return self<<value.
__lt__
__lt__(
value, /
)
Return self<value.
__mod__
__mod__(
value, /
)
Return self%value.
__mul__
__mul__(
value, /
)
Return self*value.
__ne__
__ne__(
value, /
)
Return self!=value.
__neg__
__neg__()
-self
__or__
__or__(
value, /
)
Return self|value.
__pos__
__pos__()
+self
__pow__
__pow__(
value, mod, /
)
Return pow(self, value, mod).
__radd__
__radd__(
value, /
)
Return value+self.
__rand__
__rand__(
value, /
)
Return value&self.
__rfloordiv__
__rfloordiv__(
value, /
)
Return value//self.
__rlshift__
__rlshift__(
value, /
)
Return value<<self.
__rmod__
__rmod__(
value, /
)
Return value%self.
__rmul__
__rmul__(
value, /
)
Return value*self.
__ror__
__ror__(
value, /
)
Return value|self.
__rpow__
__rpow__(
value, mod, /
)
Return pow(value, self, mod).
__rrshift__
__rrshift__(
value, /
)
Return value>>self.
__rshift__
__rshift__(
value, /
)
Return self>>value.
__rsub__
__rsub__(
value, /
)
Return value-self.
__rtruediv__
__rtruediv__(
value, /
)
Return value/self.
__rxor__
__rxor__(
value, /
)
Return value^self.
__sub__
__sub__(
value, /
)
Return self-value.
__truediv__
__truediv__(
value, /
)
Return self/value.
__xor__
__xor__(
value, /
)
Return self^value.