tf.compat.v1.variable_axis_size_partitioner
Stay organized with collections
Save and categorize content based on your preferences.
Get a partitioner for VariableScope to keep shards below max_shard_bytes.
tf.compat.v1.variable_axis_size_partitioner(
max_shard_bytes, axis=0, bytes_per_string_element=16, max_shards=None
)
This partitioner will shard a Variable along one axis, attempting to keep
the maximum shard size below max_shard_bytes. In practice, this is not
always possible when sharding along only one axis. When this happens,
this axis is sharded as much as possible (i.e., every dimension becomes
a separate shard).
If the partitioner hits the max_shards limit, then each shard may end up
larger than max_shard_bytes. By default max_shards equals None and no
limit on the number of shards is enforced.
One reasonable value for max_shard_bytes is (64 << 20) - 1, or almost
64MB, to keep below the protobuf byte limit.
Args |
|---|
max_shard_bytes
|
The maximum size any given shard is allowed to be.
|
axis
|
The axis to partition along. Default: outermost axis.
|
bytes_per_string_element
|
If the Variable is of type string, this provides
an estimate of how large each scalar in the Variable is.
|
max_shards
|
The maximum number of shards in int created taking precedence
over max_shard_bytes.
|
Returns |
|---|
A partition function usable as the partitioner argument to
variable_scope and get_variable.
|
Raises |
|---|
ValueError
|
If any of the byte counts are non-positive.
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. Some content is licensed under the numpy license.
Last updated 2024-04-26 UTC.
[null,null,["Last updated 2024-04-26 UTC."],[],[],null,["# tf.compat.v1.variable_axis_size_partitioner\n\n\u003cbr /\u003e\n\n|----------------------------------------------------------------------------------------------------------------------------------------|\n| [View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.16.1/tensorflow/python/ops/partitioned_variables.py#L67-L152) |\n\nGet a partitioner for VariableScope to keep shards below `max_shard_bytes`. \n\n tf.compat.v1.variable_axis_size_partitioner(\n max_shard_bytes, axis=0, bytes_per_string_element=16, max_shards=None\n )\n\nThis partitioner will shard a Variable along one axis, attempting to keep\nthe maximum shard size below `max_shard_bytes`. In practice, this is not\nalways possible when sharding along only one axis. When this happens,\nthis axis is sharded as much as possible (i.e., every dimension becomes\na separate shard).\n\nIf the partitioner hits the `max_shards` limit, then each shard may end up\nlarger than `max_shard_bytes`. By default `max_shards` equals `None` and no\nlimit on the number of shards is enforced.\n\nOne reasonable value for `max_shard_bytes` is `(64 \u003c\u003c 20) - 1`, or almost\n`64MB`, to keep below the protobuf byte limit.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|----------------------------|---------------------------------------------------------------------------------------------------------------|\n| `max_shard_bytes` | The maximum size any given shard is allowed to be. |\n| `axis` | The axis to partition along. Default: outermost axis. |\n| `bytes_per_string_element` | If the `Variable` is of type string, this provides an estimate of how large each scalar in the `Variable` is. |\n| `max_shards` | The maximum number of shards in int created taking precedence over `max_shard_bytes`. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ------- ||\n|---|---|\n| A partition function usable as the `partitioner` argument to `variable_scope` and `get_variable`. ||\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Raises ------ ||\n|--------------|---------------------------------------------|\n| `ValueError` | If any of the byte counts are non-positive. |\n\n\u003cbr /\u003e"]]
View source on GitHub