Parallax Strategy

Parallax StrategyBuilder.

class Parallax(chunk_size=128, local_proxy_variable=False, sync=True, staleness=0)

Bases: autodist.strategy.ps_lb_strategy.PSLoadBalancing, autodist.strategy.all_reduce_strategy.AllReduce

Generates the Parallax Strategy from https://arxiv.org/pdf/1808.02621.pdf.

The Parallax strategy mixes Parameter Server and AllReduce. The rationale is that a PS architecture is more suitable for sparse gradient updates, while AllReduce has reportedly better performance on dense gradient updates.

build(graph_item, resource_spec)

Generate the strategy.