tf.distribute.NcclAllReduce
Stay organized with collections
Save and categorize content based on your preferences.
Reduction using NCCL all-reduce.
tf.distribute.NcclAllReduce(
num_packs=1
)
Args |
|---|
num_packs
|
values will be packed in this many splits. num_packs should
be greater than or equals 0. When it is zero, no packing will be done.
|
Raises |
|---|
ValueError if num_packs is negative.
|
Methods
batch_reduce
View source
batch_reduce(
reduce_op, value_destination_pairs, experimental_hints=None
)
Reduce PerReplica objects in a batch.
Reduce each first element in value_destination_pairs to each second
element which indicates the destinations.
This can be faster than multiple individual reduces because we can
fuse several tensors into one or multiple packs before reduction.
| Args |
|---|
reduce_op
|
An instance of tf.distribute.ReduceOp that indicates how the
per_replica_value will be reduced.
|
value_destination_pairs
|
A list or a tuple of PerReplica objects (or
tensors with device set if there is one device) and destinations.
|
experimental_hints
|
A tf.distrbute.experimental.CollectiveHints. Hints
to perform collective operations.
|
| Returns |
|---|
|
a list of Mirrored objects.
|
| Raises |
|---|
ValueError
|
if value_destination_pairs is not an iterable of
tuples of PerReplica objects and destinations.
|
broadcast
View source
broadcast(
tensor, destinations
)
Broadcast the tensor to destinations.
| Args |
|---|
tensor
|
the tensor to broadcast.
|
destinations
|
the broadcast destinations.
|
| Returns |
|---|
|
a Mirrored object.
|
reduce
View source
reduce(
reduce_op, per_replica_value, destinations, experimental_hints=None
)
Reduce per_replica_value to destinations.
It runs the reduction operation defined by reduce_op and put the
result on destinations.
| Args |
|---|
reduce_op
|
An instance of tf.distribute.ReduceOp that indicates how
per_replica_value will be reduced.
|
per_replica_value
|
A PerReplica object or a tensor with device set.
|
destinations
|
the reduction destinations.
|
experimental_hints
|
A tf.distrbute.experimental.CollectiveHints. Hints
to perform collective operations.
|
| Returns |
|---|
|
a Mirrored object.
|
| Raises |
|---|
ValueError
|
if per_replica_value can't be converted to a PerReplica
object or if destinations aren't strings, Variables or DistributedValues
|
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.
Last updated 2020-10-01 UTC.
[null,null,["Last updated 2020-10-01 UTC."],[],[],null,["# tf.distribute.NcclAllReduce\n\n\u003cbr /\u003e\n\n|-------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------|\n| [TensorFlow 1 version](/versions/r1.15/api_docs/python/tf/distribute/NcclAllReduce) | [View source on GitHub](https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/python/distribute/cross_device_ops.py#L743-L765) |\n\nReduction using NCCL all-reduce.\n\n#### View aliases\n\n\n**Compat aliases for migration**\n\nSee\n[Migration guide](https://www.tensorflow.org/guide/migrate) for\nmore details.\n\n[`tf.compat.v1.distribute.NcclAllReduce`](/api_docs/python/tf/distribute/NcclAllReduce)\n\n\u003cbr /\u003e\n\n tf.distribute.NcclAllReduce(\n num_packs=1\n )\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ---- ||\n|-------------|--------------------------------------------------------------------------------------------------------------------------------------|\n| `num_packs` | values will be packed in this many splits. `num_packs` should be greater than or equals 0. When it is zero, no packing will be done. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Raises ------ ||\n|---|---|\n| ValueError if `num_packs` is negative. ||\n\n\u003cbr /\u003e\n\nMethods\n-------\n\n### `batch_reduce`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/python/distribute/cross_device_ops.py#L269-L319) \n\n batch_reduce(\n reduce_op, value_destination_pairs, experimental_hints=None\n )\n\nReduce PerReplica objects in a batch.\n\nReduce each first element in `value_destination_pairs` to each second\nelement which indicates the destinations.\n\nThis can be faster than multiple individual `reduce`s because we can\nfuse several tensors into one or multiple packs before reduction.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ||\n|---------------------------|-------------------------------------------------------------------------------------------------------------------------------------|\n| `reduce_op` | An instance of [`tf.distribute.ReduceOp`](../../tf/distribute/ReduceOp) that indicates how the `per_replica_value` will be reduced. |\n| `value_destination_pairs` | A list or a tuple of PerReplica objects (or tensors with device set if there is one device) and destinations. |\n| `experimental_hints` | A `tf.distrbute.experimental.CollectiveHints`. Hints to perform collective operations. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ||\n|---|---|\n| a list of Mirrored objects. ||\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Raises ||\n|--------------|---------------------------------------------------------------------------------------------------|\n| `ValueError` | if `value_destination_pairs` is not an iterable of tuples of PerReplica objects and destinations. |\n\n\u003cbr /\u003e\n\n### `broadcast`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/python/distribute/cross_device_ops.py#L321-L332) \n\n broadcast(\n tensor, destinations\n )\n\nBroadcast the `tensor` to destinations.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ||\n|----------------|-----------------------------|\n| `tensor` | the tensor to broadcast. |\n| `destinations` | the broadcast destinations. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ||\n|---|---|\n| a Mirrored object. ||\n\n\u003cbr /\u003e\n\n### `reduce`\n\n[View source](https://github.com/tensorflow/tensorflow/blob/v2.2.0/tensorflow/python/distribute/cross_device_ops.py#L226-L267) \n\n reduce(\n reduce_op, per_replica_value, destinations, experimental_hints=None\n )\n\nReduce `per_replica_value` to `destinations`.\n\nIt runs the reduction operation defined by `reduce_op` and put the\nresult on `destinations`.\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Args ||\n|----------------------|-------------------------------------------------------------------------------------------------------------------------------|\n| `reduce_op` | An instance of [`tf.distribute.ReduceOp`](../../tf/distribute/ReduceOp) that indicates how per_replica_value will be reduced. |\n| `per_replica_value` | A PerReplica object or a tensor with device set. |\n| `destinations` | the reduction destinations. |\n| `experimental_hints` | A `tf.distrbute.experimental.CollectiveHints`. Hints to perform collective operations. |\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Returns ||\n|---|---|\n| a Mirrored object. ||\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n\u003cbr /\u003e\n\n| Raises ||\n|--------------|----------------------------------------------------------------------------------------------------------------------------------|\n| `ValueError` | if per_replica_value can't be converted to a PerReplica object or if destinations aren't strings, Variables or DistributedValues |\n\n\u003cbr /\u003e"]]
TensorFlow 1 version
View source on GitHub