Efficient Distributed Machine Learning on Azure: Overcoming Communication Bottlenecks

Original authors: Microsoft Cloud & Information Services Lab researchers Dhruv Mahajan, Sundararajan Sellamanickam, and Keerthi Selvaraj.

Translator: Zhang Tong.

Enterprises today accumulate massive data assets such as user behavior, system access logs, and usage patterns. By leveraging cloud services like Microsoft Azure Machine Learning, companies can not only store data and perform classic retrospective business‑intelligence analysis, but also harness the cloud’s power for forward‑looking predictive analytics, gaining actionable insights that become a competitive advantage.

Collecting and maintaining big data has become a common requirement for many applications. As data volumes explode, distributed storage becomes inevitable. In many scenarios data collection is inherently distributed, leading naturally to distributed storage. Consequently, building machine‑learning solutions that process distributed data—such as logistic‑regression click‑through‑rate estimation for online advertising, deep‑learning pipelines for large image and speech datasets, or anomaly‑detection record analysis—is essential.

Efficient distributed training of machine‑learning workloads in a cluster is a key research focus of Microsoft’s Cloud & Information Services Lab (CISL). This article delves into that topic; while some details are technically deep, the core ideas are presented in an accessible manner, offering value to anyone interested in large‑scale distributed ML.

Choosing the Right Infrastructure

John Langford recently described Vowpal Wabbit (VW), a system for fast learning, and briefly discussed distributed learning on terabyte‑scale datasets. Because most ML algorithms are iterative, selecting an appropriate distributed framework to run them is critical.

MapReduce and its open‑source implementation Hadoop are popular distributed data‑processing platforms, but the heavy overhead of job scheduling, data transfer, and parsing for each iteration makes them ill‑suited for iterative ML algorithms.

A better approach is to augment the communication layer, for example by adding an All‑Reduce primitive compatible with Hadoop (as used in VW) or by adopting newer frameworks such as REEF that support efficient iterative computation.

Statistical Query Model (SQM)

The most advanced distributed‑ML algorithms today, including those in VW, are based on the Statistical Query Model (SQM). In SQM, learning proceeds by computing a function on each data point and then aggregating the results. For a linear ML problem, the result is the dot product of a feature vector and a weight‑vector, encompassing models such as logistic regression, support‑vector machines, and least‑squares fitting. Each node computes a local gradient on its data; an All‑Reduce operation then aggregates these local gradients into the global gradient.

Communication Bottleneck

Distributed computation often faces a critical bottleneck: communication bandwidth is orders of magnitude slower than raw computation. It is common for communication speed to be 10‑50× slower than compute speed.

Let T comm and T comp denote the time spent on communication and computation per iteration. The total time for an iterative ML algorithm is:

Total time = (T comm + T comp ) × #iterations

As the number of nodes grows, T comp typically decreases linearly, while T comm either rises or stays constant (assuming an efficiently implemented All‑Reduce). When the model has many parameters (d), each iteration requires communicating and updating these parameters across nodes, leading to O(d) communication cost. MapReduce exacerbates the problem because each iteration incurs a full Map‑Reduce job.

Addressing the Communication Bottleneck

Our recent research targets this bottleneck. Assuming T comm dominates, the standard SQM approach makes T comp much smaller than T comm . We ask whether it is possible to redesign the algorithm and its iterations so that T comp approaches T comm , thereby reducing the total number of iterations needed. This requires a fundamental change to the algorithm.

More Specific Details

Consider learning a linear model. In our algorithm, each node updates weights and gradients similarly to SQM, but the gradient (obtained via All‑Reduce) and local data are combined in a sophisticated way to form a local approximation of the global objective. Each node solves its own approximate problem, producing a local weight update; the collection of local updates is then merged into a global weight update. This increases per‑node computation without adding communication, so the per‑iteration time is not significantly affected, yet the number of required iterations drops dramatically. In massive datasets, each node’s local data can be sufficient for good learning, allowing our method to converge in one or two iterations where SQM might need hundreds or thousands, yielding a 2‑3× speedup on average.

We can also partition the weight vector across multiple cluster nodes, establishing a distributed storage and computation scheme where any given weight variable is updated on a single node. This is effective for scenarios such as zeroing specific weights in linear models or for distributed deep‑training, creating a special iterative algorithm that trades extra local computation for far fewer iterations.

Evaluation

The algorithms we focus on excel when communication cost is heavy, but they do not solve every problem. Numerous strong distributed‑ML algorithms exist in recent literature, yet a detailed comparative evaluation is still lacking. The best path forward remains to integrate these methods into cloud‑based ML libraries.

Automatic Distributed ML Tailored to User Needs

Another important aspect is that cloud users have diverse requirements: some prioritize fastest time‑to‑solution, others minimize cost, and some demand the most accurate result regardless of time or expense. An automated system that selects the appropriate algorithm and parameter settings based on these preferences is crucial. Our current research is focused on building such a system.

In future product development, automatic distributed machine‑learning will be a key research area for Microsoft Azure ML.