Gradient compression for communication-limited convex optimization

S. Khirirat, M. Johansson, D.-A. Alistarh, in:, 2018 IEEE Conference on Decision and Control, IEEE, 2019.

Download
No fulltext has been uploaded. References only!

Conference Paper | Published | English

Scopus indexed
Author
Khirirat, Sarit; Johansson, Mikael; Alistarh, Dan-AdrianIST Austria
Department
Abstract
Data-rich applications in machine-learning and control have motivated an intense research on large-scale optimization. Novel algorithms have been proposed and shown to have optimal convergence rates in terms of iteration counts. However, their practical performance is severely degraded by the cost of exchanging high-dimensional gradient vectors between computing nodes. Several gradient compression heuristics have recently been proposed to reduce communications, but few theoretical results exist that quantify how they impact algorithm convergence. This paper establishes and strengthens the convergence guarantees for gradient descent under a family of gradient compression techniques. For convex optimization problems, we derive admissible step sizes and quantify both the number of iterations and the number of bits that need to be exchanged to reach a target accuracy. Finally, we validate the performance of different gradient compression techniques in simulations. The numerical results highlight the properties of different gradient compression algorithms and confirm that fast convergence with limited information exchange is possible.
Publishing Year
Date Published
2019-01-21
Proceedings Title
2018 IEEE Conference on Decision and Control
Article Number
8619625
Conference
CDC: Conference on Decision and Control
Conference Location
Miami Beach, FL, United States
Conference Date
2018-12-17 – 2018-12-19
ISSN
IST-REx-ID

Cite this

Khirirat S, Johansson M, Alistarh D-A. Gradient compression for communication-limited convex optimization. In: 2018 IEEE Conference on Decision and Control. IEEE; 2019. doi:10.1109/cdc.2018.8619625
Khirirat, S., Johansson, M., & Alistarh, D.-A. (2019). Gradient compression for communication-limited convex optimization. In 2018 IEEE Conference on Decision and Control. Miami Beach, FL, United States: IEEE. https://doi.org/10.1109/cdc.2018.8619625
Khirirat, Sarit, Mikael Johansson, and Dan-Adrian Alistarh. “Gradient Compression for Communication-Limited Convex Optimization.” In 2018 IEEE Conference on Decision and Control. IEEE, 2019. https://doi.org/10.1109/cdc.2018.8619625.
S. Khirirat, M. Johansson, and D.-A. Alistarh, “Gradient compression for communication-limited convex optimization,” in 2018 IEEE Conference on Decision and Control, Miami Beach, FL, United States, 2019.
Khirirat S, Johansson M, Alistarh D-A. 2019. Gradient compression for communication-limited convex optimization. 2018 IEEE Conference on Decision and Control. CDC: Conference on Decision and Control
Khirirat, Sarit, et al. “Gradient Compression for Communication-Limited Convex Optimization.” 2018 IEEE Conference on Decision and Control, 8619625, IEEE, 2019, doi:10.1109/cdc.2018.8619625.

Export

Marked Publications

Open Data IST Research Explorer

Search this title in

Google Scholar
ISBN Search