Partitioned All Reduce Strategy¶
Partitioned PS StrategyBuilder with Greedy Load Balancer.
Partitioned AR StrategyBuilder.
This StrategyBuilder generates a strategy that partitions each variable along its first dimension, and synchronizes them using AllReduce. It might be advantageous for communicating extremely large messages – when synchronizing a single message is bounded by single-flow bandwidth.
It will also sequentially merge collective ops into a single collective group based on chunk_size. This strategy does not support synchronizing sparse updates with >1 nodes due to the TF AllGather bug.