GraphTransformer(compiled_strategy, cluster, graph_item)¶
This is the bulk of the AutoDist backend logic, taking a single-node, single-GPU graph and transforming it into a distributed graph. This all happens based on the Strategy provided.
The transformation occurs over several steps:
Partitions the necessary variables
Replicates the graph the desired number of times
Within the graph, synchronizes gradients with in-graph logic
Adds between-graph gradient synchronization logic
Call graph transformer to transform a graph item based on strategy and cluster.
Return the number of local replicas.