---
_id: '6589'
abstract:
- lang: eng
text: Distributed training of massive machine learning models, in particular deep
neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace.
Several families of communication-reduction methods, such as quantization, large-batch
methods, and gradient sparsification, have been proposed. To date, gradient sparsification
methods--where each node sorts gradients by magnitude, and only communicates a
subset of the components, accumulating the rest locally--are known to yield some
of the largest practical gains. Such methods can reduce the amount of communication
per step by up to \emph{three orders of magnitude}, while preserving model accuracy.
Yet, this family of methods currently has no theoretical justification. This is
the question we address in this paper. We prove that, under analytic assumptions,
sparsifying gradients by magnitude with local error correction provides convergence
guarantees, for both convex and non-convex smooth objectives, for data-parallel
SGD. The main insight is that sparsification methods implicitly maintain bounds
on the maximum impact of stale updates, thanks to selection by magnitude. Our
analysis and empirical validation also reveal that these methods do require analytical
conditions to converge well, justifying existing heuristics.
article_processing_charge: No
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Torsten
full_name: Hoefler, Torsten
last_name: Hoefler
- first_name: Mikael
full_name: Johansson, Mikael
last_name: Johansson
- first_name: Nikola H
full_name: Konstantinov, Nikola H
id: 4B9D76E4-F248-11E8-B48F-1D18A9856A87
last_name: Konstantinov
- first_name: Sarit
full_name: Khirirat, Sarit
last_name: Khirirat
- first_name: Cedric
full_name: Renggli, Cedric
last_name: Renggli
citation:
ama: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. The convergence of sparsified gradient methods. In: Advances in Neural Information
Processing Systems 31. Vol Volume 2018. Neural Information Processing Systems
Foundation; 2018:5973-5983.'
apa: 'Alistarh, D.-A., Hoefler, T., Johansson, M., Konstantinov, N. H., Khirirat,
S., & Renggli, C. (2018). The convergence of sparsified gradient methods.
In Advances in Neural Information Processing Systems 31 (Vol. Volume 2018,
pp. 5973–5983). Montreal, Canada: Neural Information Processing Systems Foundation.'
chicago: Alistarh, Dan-Adrian, Torsten Hoefler, Mikael Johansson, Nikola H Konstantinov,
Sarit Khirirat, and Cedric Renggli. “The Convergence of Sparsified Gradient Methods.”
In Advances in Neural Information Processing Systems 31, Volume 2018:5973–83.
Neural Information Processing Systems Foundation, 2018.
ieee: D.-A. Alistarh, T. Hoefler, M. Johansson, N. H. Konstantinov, S. Khirirat,
and C. Renggli, “The convergence of sparsified gradient methods,” in Advances
in Neural Information Processing Systems 31, Montreal, Canada, 2018, vol.
Volume 2018, pp. 5973–5983.
ista: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. 2018. The convergence of sparsified gradient methods. Advances in Neural Information
Processing Systems 31. NeurIPS: Conference on Neural Information Processing Systems
vol. Volume 2018, 5973–5983.'
mla: Alistarh, Dan-Adrian, et al. “The Convergence of Sparsified Gradient Methods.”
Advances in Neural Information Processing Systems 31, vol. Volume 2018,
Neural Information Processing Systems Foundation, 2018, pp. 5973–83.
short: D.-A. Alistarh, T. Hoefler, M. Johansson, N.H. Konstantinov, S. Khirirat,
C. Renggli, in:, Advances in Neural Information Processing Systems 31, Neural
Information Processing Systems Foundation, 2018, pp. 5973–5983.
conference:
end_date: 2018-12-08
location: Montreal, Canada
name: 'NeurIPS: Conference on Neural Information Processing Systems'
start_date: 2018-12-02
date_created: 2019-06-27T09:32:55Z
date_published: 2018-12-01T00:00:00Z
date_updated: 2023-10-17T11:47:20Z
day: '01'
department:
- _id: DaAl
- _id: ChLa
ec_funded: 1
external_id:
arxiv:
- '1809.10505'
isi:
- '000461852000047'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1809.10505
month: '12'
oa: 1
oa_version: Preprint
page: 5973-5983
project:
- _id: 2564DBCA-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '665385'
name: International IST Doctoral Program
publication: Advances in Neural Information Processing Systems 31
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: The convergence of sparsified gradient methods
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: Volume 2018
year: '2018'
...
---
_id: '7'
abstract:
- lang: eng
text: Animal social networks are shaped by multiple selection pressures, including
the need to ensure efficient communication and functioning while simultaneously
limiting disease transmission. Social animals could potentially further reduce
epidemic risk by altering their social networks in the presence of pathogens,
yet there is currently no evidence for such pathogen-triggered responses. We tested
this hypothesis experimentally in the ant Lasius niger using a combination of
automated tracking, controlled pathogen exposure, transmission quantification,
and temporally explicit simulations. Pathogen exposure induced behavioral changes
in both exposed ants and their nestmates, which helped contain the disease by
reinforcing key transmission-inhibitory properties of the colony's contact network.
This suggests that social network plasticity in response to pathogens is an effective
strategy for mitigating the effects of disease in social groups.
acknowledgement: This project was funded by two European Research Council Advanced
Grants (Social Life, 249375, and resiliANT, 741491) and two Swiss National Science
Foundation grants (CR32I3_141063 and 310030_156732) to L.K. and a European Research
Council Starting Grant (SocialVaccines, 243071) to S.C.
article_processing_charge: No
article_type: original
author:
- first_name: Nathalie
full_name: Stroeymeyt, Nathalie
last_name: Stroeymeyt
- first_name: Anna V
full_name: Grasse, Anna V
id: 406F989C-F248-11E8-B48F-1D18A9856A87
last_name: Grasse
- first_name: Alessandro
full_name: Crespi, Alessandro
last_name: Crespi
- first_name: Danielle
full_name: Mersch, Danielle
last_name: Mersch
- first_name: Sylvia
full_name: Cremer, Sylvia
id: 2F64EC8C-F248-11E8-B48F-1D18A9856A87
last_name: Cremer
orcid: 0000-0002-2193-3868
- first_name: Laurent
full_name: Keller, Laurent
last_name: Keller
citation:
ama: Stroeymeyt N, Grasse AV, Crespi A, Mersch D, Cremer S, Keller L. Social network
plasticity decreases disease transmission in a eusocial insect. Science.
2018;362(6417):941-945. doi:10.1126/science.aat4793
apa: Stroeymeyt, N., Grasse, A. V., Crespi, A., Mersch, D., Cremer, S., & Keller,
L. (2018). Social network plasticity decreases disease transmission in a eusocial
insect. Science. AAAS. https://doi.org/10.1126/science.aat4793
chicago: Stroeymeyt, Nathalie, Anna V Grasse, Alessandro Crespi, Danielle Mersch,
Sylvia Cremer, and Laurent Keller. “Social Network Plasticity Decreases Disease
Transmission in a Eusocial Insect.” Science. AAAS, 2018. https://doi.org/10.1126/science.aat4793.
ieee: N. Stroeymeyt, A. V. Grasse, A. Crespi, D. Mersch, S. Cremer, and L. Keller,
“Social network plasticity decreases disease transmission in a eusocial insect,”
Science, vol. 362, no. 6417. AAAS, pp. 941–945, 2018.
ista: Stroeymeyt N, Grasse AV, Crespi A, Mersch D, Cremer S, Keller L. 2018. Social
network plasticity decreases disease transmission in a eusocial insect. Science.
362(6417), 941–945.
mla: Stroeymeyt, Nathalie, et al. “Social Network Plasticity Decreases Disease Transmission
in a Eusocial Insect.” Science, vol. 362, no. 6417, AAAS, 2018, pp. 941–45,
doi:10.1126/science.aat4793.
short: N. Stroeymeyt, A.V. Grasse, A. Crespi, D. Mersch, S. Cremer, L. Keller, Science
362 (2018) 941–945.
date_created: 2018-12-11T11:44:07Z
date_published: 2018-11-23T00:00:00Z
date_updated: 2023-10-17T11:50:05Z
day: '23'
department:
- _id: SyCr
doi: 10.1126/science.aat4793
ec_funded: 1
external_id:
isi:
- '000451124500041'
intvolume: ' 362'
isi: 1
issue: '6417'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://serval.unil.ch/resource/serval:BIB_E9228C205467.P001/REF.pdf
month: '11'
oa: 1
oa_version: Published Version
page: 941 - 945
project:
- _id: 25DC711C-B435-11E9-9278-68D0E5697425
call_identifier: FP7
grant_number: '243071'
name: 'Social Vaccination in Ant Colonies: from Individual Mechanisms to Society
Effects'
publication: Science
publication_identifier:
issn:
- 1095-9203
publication_status: published
publisher: AAAS
publist_id: '8049'
quality_controlled: '1'
related_material:
link:
- description: News on IST Homepage
relation: press_release
url: https://ist.ac.at/en/news/for-ants-unity-is-strength-and-health/
record:
- id: '13055'
relation: research_data
status: public
scopus_import: '1'
status: public
title: Social network plasticity decreases disease transmission in a eusocial insect
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 362
year: '2018'
...
---
_id: '19'
abstract:
- lang: eng
text: Bacteria regulate genes to survive antibiotic stress, but regulation can be
far from perfect. When regulation is not optimal, mutations that change gene expression
can contribute to antibiotic resistance. It is not systematically understood to
what extent natural gene regulation is or is not optimal for distinct antibiotics,
and how changes in expression of specific genes quantitatively affect antibiotic
resistance. Here we discover a simple quantitative relation between fitness, gene
expression, and antibiotic potency, which rationalizes our observation that a
multitude of genes and even innate antibiotic defense mechanisms have expression
that is critically nonoptimal under antibiotic treatment. First, we developed
a pooled-strain drug-diffusion assay and screened Escherichia coli overexpression
and knockout libraries, finding that resistance to a range of 31 antibiotics could
result from changing expression of a large and functionally diverse set of genes,
in a primarily but not exclusively drug-specific manner. Second, by synthetically
controlling the expression of single-drug and multidrug resistance genes, we observed
that their fitness-expression functions changed dramatically under antibiotic
treatment in accordance with a log-sensitivity relation. Thus, because many genes
are nonoptimally expressed under antibiotic treatment, many regulatory mutations
can contribute to resistance by altering expression and by activating latent defenses.
article_processing_charge: No
article_type: original
author:
- first_name: Adam
full_name: Palmer, Adam
last_name: Palmer
- first_name: Remy P
full_name: Chait, Remy P
id: 3464AE84-F248-11E8-B48F-1D18A9856A87
last_name: Chait
orcid: 0000-0003-0876-3187
- first_name: Roy
full_name: Kishony, Roy
last_name: Kishony
citation:
ama: Palmer A, Chait RP, Kishony R. Nonoptimal gene expression creates latent potential
for antibiotic resistance. Molecular Biology and Evolution. 2018;35(11):2669-2684.
doi:10.1093/molbev/msy163
apa: Palmer, A., Chait, R. P., & Kishony, R. (2018). Nonoptimal gene expression
creates latent potential for antibiotic resistance. Molecular Biology and Evolution.
Oxford University Press. https://doi.org/10.1093/molbev/msy163
chicago: Palmer, Adam, Remy P Chait, and Roy Kishony. “Nonoptimal Gene Expression
Creates Latent Potential for Antibiotic Resistance.” Molecular Biology and
Evolution. Oxford University Press, 2018. https://doi.org/10.1093/molbev/msy163.
ieee: A. Palmer, R. P. Chait, and R. Kishony, “Nonoptimal gene expression creates
latent potential for antibiotic resistance,” Molecular Biology and Evolution,
vol. 35, no. 11. Oxford University Press, pp. 2669–2684, 2018.
ista: Palmer A, Chait RP, Kishony R. 2018. Nonoptimal gene expression creates latent
potential for antibiotic resistance. Molecular Biology and Evolution. 35(11),
2669–2684.
mla: Palmer, Adam, et al. “Nonoptimal Gene Expression Creates Latent Potential for
Antibiotic Resistance.” Molecular Biology and Evolution, vol. 35, no. 11,
Oxford University Press, 2018, pp. 2669–84, doi:10.1093/molbev/msy163.
short: A. Palmer, R.P. Chait, R. Kishony, Molecular Biology and Evolution 35 (2018)
2669–2684.
date_created: 2018-12-11T11:44:11Z
date_published: 2018-08-28T00:00:00Z
date_updated: 2023-10-17T11:51:06Z
day: '28'
department:
- _id: CaGu
- _id: GaTk
doi: 10.1093/molbev/msy163
external_id:
isi:
- '000452567200006'
pmid:
- '30169679'
intvolume: ' 35'
isi: 1
issue: '11'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://www.ncbi.nlm.nih.gov/pubmed/30169679
month: '08'
oa: 1
oa_version: Submitted Version
page: 2669 - 2684
pmid: 1
publication: Molecular Biology and Evolution
publication_identifier:
issn:
- 0737-4038
publication_status: published
publisher: Oxford University Press
publist_id: '8036'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Nonoptimal gene expression creates latent potential for antibiotic resistance
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2018'
...
---
_id: '13055'
abstract:
- lang: eng
text: "Dataset for manuscript 'Social network plasticity decreases disease transmission
in a eusocial insect'\r\nCompared to previous versions: - raw image files added\r\n
\ - correction of URLs within
README.txt file\r\n"
article_processing_charge: No
author:
- first_name: Nathalie
full_name: Stroeymeyt, Nathalie
last_name: Stroeymeyt
- first_name: Anna V
full_name: Grasse, Anna V
id: 406F989C-F248-11E8-B48F-1D18A9856A87
last_name: Grasse
- first_name: Alessandro
full_name: Crespi, Alessandro
last_name: Crespi
- first_name: Danielle
full_name: Mersch, Danielle
last_name: Mersch
- first_name: Sylvia
full_name: Cremer, Sylvia
id: 2F64EC8C-F248-11E8-B48F-1D18A9856A87
last_name: Cremer
orcid: 0000-0002-2193-3868
- first_name: Laurent
full_name: Keller, Laurent
last_name: Keller
citation:
ama: Stroeymeyt N, Grasse AV, Crespi A, Mersch D, Cremer S, Keller L. Social network
plasticity decreases disease transmission in a eusocial insect. 2018. doi:10.5281/ZENODO.1322669
apa: Stroeymeyt, N., Grasse, A. V., Crespi, A., Mersch, D., Cremer, S., & Keller,
L. (2018). Social network plasticity decreases disease transmission in a eusocial
insect. Zenodo. https://doi.org/10.5281/ZENODO.1322669
chicago: Stroeymeyt, Nathalie, Anna V Grasse, Alessandro Crespi, Danielle Mersch,
Sylvia Cremer, and Laurent Keller. “Social Network Plasticity Decreases Disease
Transmission in a Eusocial Insect.” Zenodo, 2018. https://doi.org/10.5281/ZENODO.1322669.
ieee: N. Stroeymeyt, A. V. Grasse, A. Crespi, D. Mersch, S. Cremer, and L. Keller,
“Social network plasticity decreases disease transmission in a eusocial insect.”
Zenodo, 2018.
ista: Stroeymeyt N, Grasse AV, Crespi A, Mersch D, Cremer S, Keller L. 2018. Social
network plasticity decreases disease transmission in a eusocial insect, Zenodo,
10.5281/ZENODO.1322669.
mla: Stroeymeyt, Nathalie, et al. Social Network Plasticity Decreases Disease
Transmission in a Eusocial Insect. Zenodo, 2018, doi:10.5281/ZENODO.1322669.
short: N. Stroeymeyt, A.V. Grasse, A. Crespi, D. Mersch, S. Cremer, L. Keller, (2018).
date_created: 2023-05-23T13:24:51Z
date_published: 2018-10-23T00:00:00Z
date_updated: 2023-10-17T11:50:04Z
day: '23'
ddc:
- '570'
department:
- _id: SyCr
doi: 10.5281/ZENODO.1322669
main_file_link:
- open_access: '1'
url: https://doi.org/10.5281/zenodo.1480665
month: '10'
oa: 1
oa_version: Published Version
publisher: Zenodo
related_material:
record:
- id: '7'
relation: used_in_publication
status: public
status: public
title: Social network plasticity decreases disease transmission in a eusocial insect
tmp:
image: /images/cc_by.png
legal_code_url: https://creativecommons.org/licenses/by/4.0/legalcode
name: Creative Commons Attribution 4.0 International Public License (CC-BY 4.0)
short: CC BY (4.0)
type: research_data_reference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
year: '2018'
...
---
_id: '22'
abstract:
- lang: eng
text: Conventional ultra-high sensitivity detectors in the millimeter-wave range
are usually cooled as their own thermal noise at room temperature would mask the
weak received radiation. The need for cryogenic systems increases the cost and
complexity of the instruments, hindering the development of, among others, airborne
and space applications. In this work, the nonlinear parametric upconversion of
millimeter-wave radiation to the optical domain inside high-quality (Q) lithium
niobate whispering-gallery mode (WGM) resonators is proposed for ultra-low noise
detection. We experimentally demonstrate coherent upconversion of millimeter-wave
signals to a 1550 nm telecom carrier, with a photon conversion efficiency surpassing
the state-of-the-art by 2 orders of magnitude. Moreover, a theoretical model shows
that the thermal equilibrium of counterpropagating WGMs is broken by overcoupling
the millimeter-wave WGM, effectively cooling the upconverted mode and allowing
ultra-low noise detection. By theoretically estimating the sensitivity of a correlation
radiometer based on the presented scheme, it is found that room-temperature radiometers
with better sensitivity than state-of-the-art high-electron-mobility transistor
(HEMT)-based radiometers can be designed. This detection paradigm can be used
to develop room-temperature instrumentation for radio astronomy, earth observation,
planetary missions, and imaging systems.
article_processing_charge: No
article_type: original
author:
- first_name: Gabriel
full_name: Botello, Gabriel
last_name: Botello
- first_name: Florian
full_name: Sedlmeir, Florian
last_name: Sedlmeir
- first_name: Alfredo R
full_name: Rueda Sanchez, Alfredo R
id: 3B82B0F8-F248-11E8-B48F-1D18A9856A87
last_name: Rueda Sanchez
orcid: 0000-0001-6249-5860
- first_name: Kerlos
full_name: Abdalmalak, Kerlos
last_name: Abdalmalak
- first_name: Elliott
full_name: Brown, Elliott
last_name: Brown
- first_name: Gerd
full_name: Leuchs, Gerd
last_name: Leuchs
- first_name: Sascha
full_name: Preu, Sascha
last_name: Preu
- first_name: Daniel
full_name: Segovia Vargas, Daniel
last_name: Segovia Vargas
- first_name: Dmitry
full_name: Strekalov, Dmitry
last_name: Strekalov
- first_name: Luis
full_name: Munoz, Luis
last_name: Munoz
- first_name: Harald
full_name: Schwefel, Harald
last_name: Schwefel
citation:
ama: Botello G, Sedlmeir F, Rueda Sanchez AR, et al. Sensitivity limits of millimeter-wave
photonic radiometers based on efficient electro-optic upconverters. Optica.
2018;5(10):1210-1219. doi:10.1364/OPTICA.5.001210
apa: Botello, G., Sedlmeir, F., Rueda Sanchez, A. R., Abdalmalak, K., Brown, E.,
Leuchs, G., … Schwefel, H. (2018). Sensitivity limits of millimeter-wave photonic
radiometers based on efficient electro-optic upconverters. Optica. https://doi.org/10.1364/OPTICA.5.001210
chicago: Botello, Gabriel, Florian Sedlmeir, Alfredo R Rueda Sanchez, Kerlos Abdalmalak,
Elliott Brown, Gerd Leuchs, Sascha Preu, et al. “Sensitivity Limits of Millimeter-Wave
Photonic Radiometers Based on Efficient Electro-Optic Upconverters.” Optica,
2018. https://doi.org/10.1364/OPTICA.5.001210.
ieee: G. Botello et al., “Sensitivity limits of millimeter-wave photonic
radiometers based on efficient electro-optic upconverters,” Optica, vol.
5, no. 10. pp. 1210–1219, 2018.
ista: Botello G, Sedlmeir F, Rueda Sanchez AR, Abdalmalak K, Brown E, Leuchs G,
Preu S, Segovia Vargas D, Strekalov D, Munoz L, Schwefel H. 2018. Sensitivity
limits of millimeter-wave photonic radiometers based on efficient electro-optic
upconverters. Optica. 5(10), 1210–1219.
mla: Botello, Gabriel, et al. “Sensitivity Limits of Millimeter-Wave Photonic Radiometers
Based on Efficient Electro-Optic Upconverters.” Optica, vol. 5, no. 10,
2018, pp. 1210–19, doi:10.1364/OPTICA.5.001210.
short: G. Botello, F. Sedlmeir, A.R. Rueda Sanchez, K. Abdalmalak, E. Brown, G.
Leuchs, S. Preu, D. Segovia Vargas, D. Strekalov, L. Munoz, H. Schwefel, Optica
5 (2018) 1210–1219.
date_created: 2018-12-11T11:44:12Z
date_published: 2018-10-20T00:00:00Z
date_updated: 2023-10-17T12:12:40Z
day: '20'
department:
- _id: JoFi
doi: 10.1364/OPTICA.5.001210
external_id:
isi:
- '000447853100007'
intvolume: ' 5'
isi: 1
issue: '10'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: 'www.doi.org/10.1364/OPTICA.5.001210 '
month: '10'
oa: 1
oa_version: Published Version
page: 1210 - 1219
publication: Optica
publication_identifier:
issn:
- '23342536'
publication_status: published
publist_id: '8033'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Sensitivity limits of millimeter-wave photonic radiometers based on efficient
electro-optic upconverters
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 5
year: '2018'
...