Synchronizer

Synchronizer.

class Synchronizer[source]

Bases: abc.ABC

Synchronizer.

Given a variable, can modify the TF Graph to synchronize its gradients in either an in-graph or a between-graph fashion.

  • In-graph means the synchronization happens in one tf.Graph

  • Between-graph means the synchronization happens across

    multiple tf.Graphs (e.g., each worker has its own graph)

assign_cluster_information(num_workers, num_replicas, worker_device, worker_id, canonical_replica_devices, is_chief=False)[source]

Store cluster information in the synchronizer.

abstract in_graph_apply(graph_item, var_name)[source]

Apply in-graph synchronization to the grad and target in the graph.

Parameters

  • graph_item (graph_item.GraphItem) – The graph to put the new ops in.

  • var_name (str) – The variable name w/o the replica prefix.

Returns

graph_item.GraphItem

abstract between_graph_apply(graph_item, var_name)[source]

Apply between-graph synchronization to the target ops in the graph.

Parameters

  • graph_item (graph_item.GraphItem) – The graph to put the new ops in.

  • var_name (str) – The variable name w/o the replica prefix.

Returns

graph_item.GraphItem

classmethod create(name, *args, **kwargs)[source]

Create new Synchronizer instance given subclass name.

Parameters

  • name – Name of the Synchronizer subclass (e.g. PSSynchronizer).

  • *args – Any args for the subclass constructor.

  • **kwargs – Any kwargs for the subclass constructor.

Returns

Synchronizer