All Reduce Strategy¶

AllReduce StrategyBuilder.

class AllReduce(chunk_size=128, all_reduce_spec='NCCL', compressor='NoneCompressor')[source]¶

Bases: autodist.strategy.base.StrategyBuilder

AllReduce StrategyBuilder.

This StrategyBuilder generates a strategy that synchronizes every dense variable using AllReduce. It will sequentially merge collective ops into a single collective group based on chunk_size.

This strategy does not support synchronizing sparse updates with >1 nodes due to the TF AllGather bug.

build(graph_item, resource_spec)[source]¶: Generate the strategy.