All Reduce Strategy¶
AllReduce(chunk_size=128, all_reduce_spec='NCCL', compressor='NoneCompressor')¶
This StrategyBuilder generates a strategy that synchronizes every dense variable using AllReduce. It will sequentially merge collective ops into a single collective group based on chunk_size.
This strategy does not support synchronizing sparse updates with >1 nodes due to the TF AllGather bug.
Generate the strategy.