All Reduce Synchronizer¶
This AllReduce Synchronizer currently uses TensorFlow’s collective_device_ops to insert their AllReduce ops into our graph.
The class AllReduceSynchronizer class contains the following possible instantiations:
spec=`auto`: single-node multiple devices, or cross-node AllReduce based on collective ops
spec=`nccl`: single-node multiple devices, or cross-node AllReduce based on NCCL
spec=`ring`/’tree’, AllReduce with different reduction structures: ring, tree, etc.
However note that it does not contain the following instantiations:
shuffle reduce (reduce to CPU or GPU as in PS) + AllReduce across nodes
any other types of hybrid reduction of PS and AllReduce.
Perform in-graph synchronization based on AllReduce and TensorFlow Collective Ops.
Note that collective ops now only supports dense tensors.
assign_cluster_information(num_workers, num_replicas, worker_device, worker_id, canonical_replica_devices, is_chief=False)¶
Store cluster information in the synchronizer.
Allreduce synchronizer will do nothing in between-graph synchronization.
create(name, *args, **kwargs)¶
Create new Synchronizer instance given subclass name.
name – Name of the Synchronizer subclass (e.g. PSSynchronizer).
*args – Any args for the subclass constructor.
**kwargs – Any kwargs for the subclass constructor.