Compressor

Gradient Compressors for All-Reduce.

class CollectiveOpsConfig

Bases: object

Config for using Collective Ops.

group_size: int

group_key: str

instance_key: str

merge_op: str

final_op: str

class Compressor(var_op_name)

Bases: abc.ABC

Wraps CollectiveOps.All_Reduce with compression and decompression for network efficiency.

This means that it only wraps gradient transmission for AllReduce synchronized variables, not PS ops or other ops like network reads.

abstract reduce(tensor: tensorflow.python.framework.ops.Tensor, conf: autodist.kernel.synchronization.compressor.CollectiveOpsConfig)

Compress, reduce, and decompress a given tensor.

Parameters

• tensor (Tensor) – the Tensor to reduce.

• conf (CollectiveOpsConfig) – the config for Collective Ops.

Returns

Reduced Tensor

classmethod create(name, *args, **kwargs)

Create new Compressor instance given subclass name.

Parameters

• name – Name of the Compressor subclass (e.g. NoneCompressor).

• *args – Any args for the subclass constructor.

• **kwargs – Any kwargs for the subclass constructor.

Returns

Compressor

class CompressorEF(var_op_name)

Bases: autodist.kernel.synchronization.compressor.Compressor, abc.ABC

A Compressor with Error Feedback.

reduce(tensor: tensorflow.python.framework.ops.Tensor, conf: autodist.kernel.synchronization.compressor.CollectiveOpsConfig)

Compress, reduce, and decompress a given tensor.

Parameters

• tensor (Tensor) – the Tensor to reduce.

• conf (CollectiveOpsConfig) – the config for Collective Ops.

Returns

Reduced Tensor

classmethod create(name, *args, **kwargs)

Create new Compressor instance given subclass name.

Parameters

• name – Name of the Compressor subclass (e.g. NoneCompressor).

• *args – Any args for the subclass constructor.

• **kwargs – Any kwargs for the subclass constructor.

Returns

Compressor

class NoneCompressor(var_op_name)

Bases: autodist.kernel.synchronization.compressor.Compressor

An identity Compressor.

reduce(tensor: tensorflow.python.framework.ops.Tensor, conf: autodist.kernel.synchronization.compressor.CollectiveOpsConfig)

Compress, reduce, and decompress a given tensor.

Parameters

• tensor (Tensor) – the Tensor to reduce.

• conf (CollectiveOpsConfig) – the config for Collective Ops.

Returns

Reduced Tensor

classmethod create(name, *args, **kwargs)

Create new Compressor instance given subclass name.

Parameters

• name – Name of the Compressor subclass (e.g. NoneCompressor).

• *args – Any args for the subclass constructor.

• **kwargs – Any kwargs for the subclass constructor.

Returns

Compressor

class HorovodCompressor(var_op_name)

Bases: autodist.kernel.synchronization.compressor.Compressor

Implements Horovod’s Compression.

reduce(tensor: tensorflow.python.framework.ops.Tensor, conf: autodist.kernel.synchronization.compressor.CollectiveOpsConfig)

Compress, reduce, and decompress a given tensor.

Parameters

• tensor (Tensor) – the Tensor to reduce.

• conf (CollectiveOpsConfig) – the config for Collective Ops.

Returns

Reduced Tensor

classmethod create(name, *args, **kwargs)

Create new Compressor instance given subclass name.

Parameters

• name – Name of the Compressor subclass (e.g. NoneCompressor).

• *args – Any args for the subclass constructor.

• **kwargs – Any kwargs for the subclass constructor.

Returns

Compressor

class HorovodCompressorEF(var_op_name)

Bases: autodist.kernel.synchronization.compressor.CompressorEF, autodist.kernel.synchronization.compressor.HorovodCompressor

Horovod’s Compression but with Error Feedback.

classmethod create(name, *args, **kwargs)

Create new Compressor instance given subclass name.

Parameters

• name – Name of the Compressor subclass (e.g. NoneCompressor).

• *args – Any args for the subclass constructor.

• **kwargs – Any kwargs for the subclass constructor.

Returns

Compressor

reduce(tensor: tensorflow.python.framework.ops.Tensor, conf: autodist.kernel.synchronization.compressor.CollectiveOpsConfig)

Compress, reduce, and decompress a given tensor.

Parameters

• tensor (Tensor) – the Tensor to reduce.

• conf (CollectiveOpsConfig) – the config for Collective Ops.

Returns

Reduced Tensor