Does AutoDist support heterogeneous clusters?
Yes, it’s as straight-forward as the setup on a homogeneous cluster. You can refer to Train on Multiple Nodes.
What device types does AutoDist support?
Currently AutoDist only supports strategies on
GPUto be configured in the [resource specification]. But we are still actively improving this.
Why doesn’t AutoDist support Eager mode?
AutoDist’s design is based on transforming / compiling the computation graph. Although eager mode can still utilize
tf.functionfor graph execution, it requires more effort to deal with variable states under eager mode to fit in the AutoDist stack, so we de-prioritized it. We might support it in the future, though!
Will AutoDist support PyTorch?
The current architecture of AutoDist is based on graph transformations. At this time, PyTorch does not offer a good way to get a static computational graph directly (except TorchScript, which is still in an early stage), so we thus do not have plan to integrate the stack with PyTorch in the near future.
Will there be Kubernetes integration?
AutoDist is integrated with Kubernetes internally in a Petuum closed-source product. Even so, one can still start with our Running on Docker instructions for containerization and orchestration.
Does AutoDist support model parallelism?