A brief tutorial on distributed and concurrent machine learning

D.-A. Alistarh, in:, Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing  - PODC ’18, ACM Press, 2018, pp. 487–488.

Download
No fulltext has been uploaded. References only!

Conference Paper | Published | English
Department
Abstract
The area of machine learning has made considerable progress over the past decade, enabled by the widespread availability of large datasets, as well as by improved algorithms and models. Given the large computational demands of machine learning workloads, parallelism, implemented either through single-node concurrency or through multi-node distribution, has been a third key ingredient to advances in machine learning. The goal of this tutorial is to provide the audience with an overview of standard distribution techniques in machine learning, with an eye towards the intriguing trade-offs between synchronization and communication costs of distributed machine learning algorithms, on the one hand, and their convergence, on the other.The tutorial will focus on parallelization strategies for the fundamental stochastic gradient descent (SGD) algorithm, which is a key tool when training machine learning models, from classical instances such as linear regression, to state-of-the-art neural network architectures. The tutorial will describe the guarantees provided by this algorithm in the sequential case, and then move on to cover both shared-memory and message-passing parallelization strategies, together with the guarantees they provide, and corresponding trade-offs. The presentation will conclude with a broad overview of ongoing research in distributed and concurrent machine learning. The tutorial will assume no prior knowledge beyond familiarity with basic concepts in algebra and analysis.
Publishing Year
Date Published
2018-07-27
Proceedings Title
Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing - PODC '18
Page
487-488
Conference
PODC: Principles of Distributed Computing
Conference Location
Egham, United Kingdom
Conference Date
2018-07-23 – 2018-07-27
IST-REx-ID

Cite this

Alistarh D-A. A brief tutorial on distributed and concurrent machine learning. In: Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing  - PODC ’18. ACM Press; 2018:487-488. doi:10.1145/3212734.3212798
Alistarh, D.-A. (2018). A brief tutorial on distributed and concurrent machine learning. In Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing  - PODC ’18 (pp. 487–488). Egham, United Kingdom: ACM Press. https://doi.org/10.1145/3212734.3212798
Alistarh, Dan-Adrian. “A Brief Tutorial on Distributed and Concurrent Machine Learning.” In Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing  - PODC ’18, 487–88. ACM Press, 2018. https://doi.org/10.1145/3212734.3212798.
D.-A. Alistarh, “A brief tutorial on distributed and concurrent machine learning,” in Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing  - PODC ’18, Egham, United Kingdom, 2018, pp. 487–488.
Alistarh D-A. 2018. A brief tutorial on distributed and concurrent machine learning. Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing  - PODC ’18. PODC: Principles of Distributed Computing 487–488.
Alistarh, Dan-Adrian. “A Brief Tutorial on Distributed and Concurrent Machine Learning.” Proceedings of the 2018 ACM Symposium on Principles of Distributed Computing  - PODC ’18, ACM Press, 2018, pp. 487–88, doi:10.1145/3212734.3212798.

Export

Marked Publications

Open Data IST Research Explorer

Search this title in

Google Scholar
ISBN Search