---
_id: '6012'
abstract:
- lang: eng
text: We present an approach to identify concise equations from data using a shallow
neural network approach. In contrast to ordinary black-box regression, this approach
allows understanding functional relations and generalizing them from observed
data to unseen parts of the parameter space. We show how to extend the class of
learnable equations for a recently proposed equation learning network to include
divisions, and we improve the learning and model selection strategy to be useful
for challenging real-world data. For systems governed by analytical expressions,
our method can in many cases identify the true underlying equation and extrapolate
to unseen domains. We demonstrate its effectiveness by experiments on a cart-pendulum
system, where only 2 random rollouts are required to learn the forward dynamics
and successfully achieve the swing-up task.
article_processing_charge: No
author:
- first_name: Subham
full_name: Sahoo, Subham
last_name: Sahoo
- first_name: Christoph
full_name: Lampert, Christoph
id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
last_name: Lampert
orcid: 0000-0001-8622-7887
- first_name: Georg S
full_name: Martius, Georg S
id: 3A276B68-F248-11E8-B48F-1D18A9856A87
last_name: Martius
citation:
ama: 'Sahoo S, Lampert C, Martius GS. Learning equations for extrapolation and control.
In: Proceedings of the 35th International Conference on Machine Learning.
Vol 80. ML Research Press; 2018:4442-4450.'
apa: 'Sahoo, S., Lampert, C., & Martius, G. S. (2018). Learning equations for
extrapolation and control. In Proceedings of the 35th International Conference
on Machine Learning (Vol. 80, pp. 4442–4450). Stockholm, Sweden: ML Research
Press.'
chicago: Sahoo, Subham, Christoph Lampert, and Georg S Martius. “Learning Equations
for Extrapolation and Control.” In Proceedings of the 35th International Conference
on Machine Learning, 80:4442–50. ML Research Press, 2018.
ieee: S. Sahoo, C. Lampert, and G. S. Martius, “Learning equations for extrapolation
and control,” in Proceedings of the 35th International Conference on Machine
Learning, Stockholm, Sweden, 2018, vol. 80, pp. 4442–4450.
ista: 'Sahoo S, Lampert C, Martius GS. 2018. Learning equations for extrapolation
and control. Proceedings of the 35th International Conference on Machine Learning.
ICML: International Conference on Machine Learning vol. 80, 4442–4450.'
mla: Sahoo, Subham, et al. “Learning Equations for Extrapolation and Control.” Proceedings
of the 35th International Conference on Machine Learning, vol. 80, ML Research
Press, 2018, pp. 4442–50.
short: S. Sahoo, C. Lampert, G.S. Martius, in:, Proceedings of the 35th International
Conference on Machine Learning, ML Research Press, 2018, pp. 4442–4450.
conference:
end_date: 2018-07-15
location: Stockholm, Sweden
name: 'ICML: International Conference on Machine Learning'
start_date: 2018-07-10
date_created: 2019-02-14T15:21:07Z
date_published: 2018-02-01T00:00:00Z
date_updated: 2023-10-17T09:50:53Z
day: '01'
department:
- _id: ChLa
ec_funded: 1
external_id:
arxiv:
- '1806.07259'
isi:
- '000683379204058'
intvolume: ' 80'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1806.07259
month: '02'
oa: 1
oa_version: Preprint
page: 4442-4450
project:
- _id: 25681D80-B435-11E9-9278-68D0E5697425
call_identifier: FP7
grant_number: '291734'
name: International IST Postdoc Fellowship Programme
publication: Proceedings of the 35th International Conference on Machine Learning
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
related_material:
link:
- description: News on IST Homepage
relation: press_release
url: https://ist.ac.at/en/news/first-machine-learning-method-capable-of-accurate-extrapolation/
scopus_import: '1'
status: public
title: Learning equations for extrapolation and control
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 80
year: '2018'
...
---
_id: '6011'
abstract:
- lang: eng
text: 'We establish a data-dependent notion of algorithmic stability for Stochastic
Gradient Descent (SGD), and employ it to develop novel generalization bounds.
This is in contrast to previous distribution-free algorithmic stability results
for SGD which depend on the worst-case constants. By virtue of the data-dependent
argument, our bounds provide new insights into learning with SGD on convex and
non-convex problems. In the convex case, we show that the bound on the generalization
error depends on the risk at the initialization point. In the non-convex case,
we prove that the expected curvature of the objective function around the initialization
point has crucial influence on the generalization error. In both cases, our results
suggest a simple data-driven strategy to stabilize SGD by pre-screening its initialization.
As a corollary, our results allow us to show optimistic generalization bounds
that exhibit fast convergence rates for SGD subject to a vanishing empirical risk
and low noise of stochastic gradient. '
article_processing_charge: No
author:
- first_name: Ilja
full_name: Kuzborskij, Ilja
last_name: Kuzborskij
- first_name: Christoph
full_name: Lampert, Christoph
id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
last_name: Lampert
orcid: 0000-0001-8622-7887
citation:
ama: 'Kuzborskij I, Lampert C. Data-dependent stability of stochastic gradient descent.
In: Proceedings of the 35 Th International Conference on Machine Learning.
Vol 80. ML Research Press; 2018:2815-2824.'
apa: 'Kuzborskij, I., & Lampert, C. (2018). Data-dependent stability of stochastic
gradient descent. In Proceedings of the 35 th International Conference on Machine
Learning (Vol. 80, pp. 2815–2824). Stockholm, Sweden: ML Research Press.'
chicago: Kuzborskij, Ilja, and Christoph Lampert. “Data-Dependent Stability of Stochastic
Gradient Descent.” In Proceedings of the 35 Th International Conference on
Machine Learning, 80:2815–24. ML Research Press, 2018.
ieee: I. Kuzborskij and C. Lampert, “Data-dependent stability of stochastic gradient
descent,” in Proceedings of the 35 th International Conference on Machine Learning,
Stockholm, Sweden, 2018, vol. 80, pp. 2815–2824.
ista: 'Kuzborskij I, Lampert C. 2018. Data-dependent stability of stochastic gradient
descent. Proceedings of the 35 th International Conference on Machine Learning.
ICML: International Conference on Machine Learning vol. 80, 2815–2824.'
mla: Kuzborskij, Ilja, and Christoph Lampert. “Data-Dependent Stability of Stochastic
Gradient Descent.” Proceedings of the 35 Th International Conference on Machine
Learning, vol. 80, ML Research Press, 2018, pp. 2815–24.
short: I. Kuzborskij, C. Lampert, in:, Proceedings of the 35 Th International Conference
on Machine Learning, ML Research Press, 2018, pp. 2815–2824.
conference:
end_date: 2018-07-15
location: Stockholm, Sweden
name: 'ICML: International Conference on Machine Learning'
start_date: 2018-07-10
date_created: 2019-02-14T14:51:57Z
date_published: 2018-02-01T00:00:00Z
date_updated: 2023-10-17T09:51:13Z
day: '01'
department:
- _id: ChLa
ec_funded: 1
external_id:
arxiv:
- '1703.01678'
isi:
- '000683379202095'
intvolume: ' 80'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1703.01678
month: '02'
oa: 1
oa_version: Preprint
page: 2815-2824
project:
- _id: 2532554C-B435-11E9-9278-68D0E5697425
call_identifier: FP7
grant_number: '308036'
name: Lifelong Learning of Visual Scene Understanding
publication: Proceedings of the 35 th International Conference on Machine Learning
publication_status: published
publisher: ML Research Press
quality_controlled: '1'
scopus_import: '1'
status: public
title: Data-dependent stability of stochastic gradient descent
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 80
year: '2018'
...
---
_id: '6589'
abstract:
- lang: eng
text: Distributed training of massive machine learning models, in particular deep
neural networks, via Stochastic Gradient Descent (SGD) is becoming commonplace.
Several families of communication-reduction methods, such as quantization, large-batch
methods, and gradient sparsification, have been proposed. To date, gradient sparsification
methods--where each node sorts gradients by magnitude, and only communicates a
subset of the components, accumulating the rest locally--are known to yield some
of the largest practical gains. Such methods can reduce the amount of communication
per step by up to \emph{three orders of magnitude}, while preserving model accuracy.
Yet, this family of methods currently has no theoretical justification. This is
the question we address in this paper. We prove that, under analytic assumptions,
sparsifying gradients by magnitude with local error correction provides convergence
guarantees, for both convex and non-convex smooth objectives, for data-parallel
SGD. The main insight is that sparsification methods implicitly maintain bounds
on the maximum impact of stale updates, thanks to selection by magnitude. Our
analysis and empirical validation also reveal that these methods do require analytical
conditions to converge well, justifying existing heuristics.
article_processing_charge: No
author:
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Torsten
full_name: Hoefler, Torsten
last_name: Hoefler
- first_name: Mikael
full_name: Johansson, Mikael
last_name: Johansson
- first_name: Nikola H
full_name: Konstantinov, Nikola H
id: 4B9D76E4-F248-11E8-B48F-1D18A9856A87
last_name: Konstantinov
- first_name: Sarit
full_name: Khirirat, Sarit
last_name: Khirirat
- first_name: Cedric
full_name: Renggli, Cedric
last_name: Renggli
citation:
ama: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. The convergence of sparsified gradient methods. In: Advances in Neural Information
Processing Systems 31. Vol Volume 2018. Neural Information Processing Systems
Foundation; 2018:5973-5983.'
apa: 'Alistarh, D.-A., Hoefler, T., Johansson, M., Konstantinov, N. H., Khirirat,
S., & Renggli, C. (2018). The convergence of sparsified gradient methods.
In Advances in Neural Information Processing Systems 31 (Vol. Volume 2018,
pp. 5973–5983). Montreal, Canada: Neural Information Processing Systems Foundation.'
chicago: Alistarh, Dan-Adrian, Torsten Hoefler, Mikael Johansson, Nikola H Konstantinov,
Sarit Khirirat, and Cedric Renggli. “The Convergence of Sparsified Gradient Methods.”
In Advances in Neural Information Processing Systems 31, Volume 2018:5973–83.
Neural Information Processing Systems Foundation, 2018.
ieee: D.-A. Alistarh, T. Hoefler, M. Johansson, N. H. Konstantinov, S. Khirirat,
and C. Renggli, “The convergence of sparsified gradient methods,” in Advances
in Neural Information Processing Systems 31, Montreal, Canada, 2018, vol.
Volume 2018, pp. 5973–5983.
ista: 'Alistarh D-A, Hoefler T, Johansson M, Konstantinov NH, Khirirat S, Renggli
C. 2018. The convergence of sparsified gradient methods. Advances in Neural Information
Processing Systems 31. NeurIPS: Conference on Neural Information Processing Systems
vol. Volume 2018, 5973–5983.'
mla: Alistarh, Dan-Adrian, et al. “The Convergence of Sparsified Gradient Methods.”
Advances in Neural Information Processing Systems 31, vol. Volume 2018,
Neural Information Processing Systems Foundation, 2018, pp. 5973–83.
short: D.-A. Alistarh, T. Hoefler, M. Johansson, N.H. Konstantinov, S. Khirirat,
C. Renggli, in:, Advances in Neural Information Processing Systems 31, Neural
Information Processing Systems Foundation, 2018, pp. 5973–5983.
conference:
end_date: 2018-12-08
location: Montreal, Canada
name: 'NeurIPS: Conference on Neural Information Processing Systems'
start_date: 2018-12-02
date_created: 2019-06-27T09:32:55Z
date_published: 2018-12-01T00:00:00Z
date_updated: 2023-10-17T11:47:20Z
day: '01'
department:
- _id: DaAl
- _id: ChLa
ec_funded: 1
external_id:
arxiv:
- '1809.10505'
isi:
- '000461852000047'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1809.10505
month: '12'
oa: 1
oa_version: Preprint
page: 5973-5983
project:
- _id: 2564DBCA-B435-11E9-9278-68D0E5697425
call_identifier: H2020
grant_number: '665385'
name: International IST Doctoral Program
publication: Advances in Neural Information Processing Systems 31
publication_status: published
publisher: Neural Information Processing Systems Foundation
quality_controlled: '1'
scopus_import: '1'
status: public
title: The convergence of sparsified gradient methods
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: Volume 2018
year: '2018'
...
---
_id: '7'
abstract:
- lang: eng
text: Animal social networks are shaped by multiple selection pressures, including
the need to ensure efficient communication and functioning while simultaneously
limiting disease transmission. Social animals could potentially further reduce
epidemic risk by altering their social networks in the presence of pathogens,
yet there is currently no evidence for such pathogen-triggered responses. We tested
this hypothesis experimentally in the ant Lasius niger using a combination of
automated tracking, controlled pathogen exposure, transmission quantification,
and temporally explicit simulations. Pathogen exposure induced behavioral changes
in both exposed ants and their nestmates, which helped contain the disease by
reinforcing key transmission-inhibitory properties of the colony's contact network.
This suggests that social network plasticity in response to pathogens is an effective
strategy for mitigating the effects of disease in social groups.
acknowledgement: This project was funded by two European Research Council Advanced
Grants (Social Life, 249375, and resiliANT, 741491) and two Swiss National Science
Foundation grants (CR32I3_141063 and 310030_156732) to L.K. and a European Research
Council Starting Grant (SocialVaccines, 243071) to S.C.
article_processing_charge: No
article_type: original
author:
- first_name: Nathalie
full_name: Stroeymeyt, Nathalie
last_name: Stroeymeyt
- first_name: Anna V
full_name: Grasse, Anna V
id: 406F989C-F248-11E8-B48F-1D18A9856A87
last_name: Grasse
- first_name: Alessandro
full_name: Crespi, Alessandro
last_name: Crespi
- first_name: Danielle
full_name: Mersch, Danielle
last_name: Mersch
- first_name: Sylvia
full_name: Cremer, Sylvia
id: 2F64EC8C-F248-11E8-B48F-1D18A9856A87
last_name: Cremer
orcid: 0000-0002-2193-3868
- first_name: Laurent
full_name: Keller, Laurent
last_name: Keller
citation:
ama: Stroeymeyt N, Grasse AV, Crespi A, Mersch D, Cremer S, Keller L. Social network
plasticity decreases disease transmission in a eusocial insect. Science.
2018;362(6417):941-945. doi:10.1126/science.aat4793
apa: Stroeymeyt, N., Grasse, A. V., Crespi, A., Mersch, D., Cremer, S., & Keller,
L. (2018). Social network plasticity decreases disease transmission in a eusocial
insect. Science. AAAS. https://doi.org/10.1126/science.aat4793
chicago: Stroeymeyt, Nathalie, Anna V Grasse, Alessandro Crespi, Danielle Mersch,
Sylvia Cremer, and Laurent Keller. “Social Network Plasticity Decreases Disease
Transmission in a Eusocial Insect.” Science. AAAS, 2018. https://doi.org/10.1126/science.aat4793.
ieee: N. Stroeymeyt, A. V. Grasse, A. Crespi, D. Mersch, S. Cremer, and L. Keller,
“Social network plasticity decreases disease transmission in a eusocial insect,”
Science, vol. 362, no. 6417. AAAS, pp. 941–945, 2018.
ista: Stroeymeyt N, Grasse AV, Crespi A, Mersch D, Cremer S, Keller L. 2018. Social
network plasticity decreases disease transmission in a eusocial insect. Science.
362(6417), 941–945.
mla: Stroeymeyt, Nathalie, et al. “Social Network Plasticity Decreases Disease Transmission
in a Eusocial Insect.” Science, vol. 362, no. 6417, AAAS, 2018, pp. 941–45,
doi:10.1126/science.aat4793.
short: N. Stroeymeyt, A.V. Grasse, A. Crespi, D. Mersch, S. Cremer, L. Keller, Science
362 (2018) 941–945.
date_created: 2018-12-11T11:44:07Z
date_published: 2018-11-23T00:00:00Z
date_updated: 2023-10-17T11:50:05Z
day: '23'
department:
- _id: SyCr
doi: 10.1126/science.aat4793
ec_funded: 1
external_id:
isi:
- '000451124500041'
intvolume: ' 362'
isi: 1
issue: '6417'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://serval.unil.ch/resource/serval:BIB_E9228C205467.P001/REF.pdf
month: '11'
oa: 1
oa_version: Published Version
page: 941 - 945
project:
- _id: 25DC711C-B435-11E9-9278-68D0E5697425
call_identifier: FP7
grant_number: '243071'
name: 'Social Vaccination in Ant Colonies: from Individual Mechanisms to Society
Effects'
publication: Science
publication_identifier:
issn:
- 1095-9203
publication_status: published
publisher: AAAS
publist_id: '8049'
quality_controlled: '1'
related_material:
link:
- description: News on IST Homepage
relation: press_release
url: https://ist.ac.at/en/news/for-ants-unity-is-strength-and-health/
record:
- id: '13055'
relation: research_data
status: public
scopus_import: '1'
status: public
title: Social network plasticity decreases disease transmission in a eusocial insect
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 362
year: '2018'
...
---
_id: '19'
abstract:
- lang: eng
text: Bacteria regulate genes to survive antibiotic stress, but regulation can be
far from perfect. When regulation is not optimal, mutations that change gene expression
can contribute to antibiotic resistance. It is not systematically understood to
what extent natural gene regulation is or is not optimal for distinct antibiotics,
and how changes in expression of specific genes quantitatively affect antibiotic
resistance. Here we discover a simple quantitative relation between fitness, gene
expression, and antibiotic potency, which rationalizes our observation that a
multitude of genes and even innate antibiotic defense mechanisms have expression
that is critically nonoptimal under antibiotic treatment. First, we developed
a pooled-strain drug-diffusion assay and screened Escherichia coli overexpression
and knockout libraries, finding that resistance to a range of 31 antibiotics could
result from changing expression of a large and functionally diverse set of genes,
in a primarily but not exclusively drug-specific manner. Second, by synthetically
controlling the expression of single-drug and multidrug resistance genes, we observed
that their fitness-expression functions changed dramatically under antibiotic
treatment in accordance with a log-sensitivity relation. Thus, because many genes
are nonoptimally expressed under antibiotic treatment, many regulatory mutations
can contribute to resistance by altering expression and by activating latent defenses.
article_processing_charge: No
article_type: original
author:
- first_name: Adam
full_name: Palmer, Adam
last_name: Palmer
- first_name: Remy P
full_name: Chait, Remy P
id: 3464AE84-F248-11E8-B48F-1D18A9856A87
last_name: Chait
orcid: 0000-0003-0876-3187
- first_name: Roy
full_name: Kishony, Roy
last_name: Kishony
citation:
ama: Palmer A, Chait RP, Kishony R. Nonoptimal gene expression creates latent potential
for antibiotic resistance. Molecular Biology and Evolution. 2018;35(11):2669-2684.
doi:10.1093/molbev/msy163
apa: Palmer, A., Chait, R. P., & Kishony, R. (2018). Nonoptimal gene expression
creates latent potential for antibiotic resistance. Molecular Biology and Evolution.
Oxford University Press. https://doi.org/10.1093/molbev/msy163
chicago: Palmer, Adam, Remy P Chait, and Roy Kishony. “Nonoptimal Gene Expression
Creates Latent Potential for Antibiotic Resistance.” Molecular Biology and
Evolution. Oxford University Press, 2018. https://doi.org/10.1093/molbev/msy163.
ieee: A. Palmer, R. P. Chait, and R. Kishony, “Nonoptimal gene expression creates
latent potential for antibiotic resistance,” Molecular Biology and Evolution,
vol. 35, no. 11. Oxford University Press, pp. 2669–2684, 2018.
ista: Palmer A, Chait RP, Kishony R. 2018. Nonoptimal gene expression creates latent
potential for antibiotic resistance. Molecular Biology and Evolution. 35(11),
2669–2684.
mla: Palmer, Adam, et al. “Nonoptimal Gene Expression Creates Latent Potential for
Antibiotic Resistance.” Molecular Biology and Evolution, vol. 35, no. 11,
Oxford University Press, 2018, pp. 2669–84, doi:10.1093/molbev/msy163.
short: A. Palmer, R.P. Chait, R. Kishony, Molecular Biology and Evolution 35 (2018)
2669–2684.
date_created: 2018-12-11T11:44:11Z
date_published: 2018-08-28T00:00:00Z
date_updated: 2023-10-17T11:51:06Z
day: '28'
department:
- _id: CaGu
- _id: GaTk
doi: 10.1093/molbev/msy163
external_id:
isi:
- '000452567200006'
pmid:
- '30169679'
intvolume: ' 35'
isi: 1
issue: '11'
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://www.ncbi.nlm.nih.gov/pubmed/30169679
month: '08'
oa: 1
oa_version: Submitted Version
page: 2669 - 2684
pmid: 1
publication: Molecular Biology and Evolution
publication_identifier:
issn:
- 0737-4038
publication_status: published
publisher: Oxford University Press
publist_id: '8036'
quality_controlled: '1'
scopus_import: '1'
status: public
title: Nonoptimal gene expression creates latent potential for antibiotic resistance
type: journal_article
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: 35
year: '2018'
...