---
_id: '1000'
abstract:
- lang: eng
text: 'We study probabilistic models of natural images and extend the autoregressive
family of PixelCNN models by incorporating latent variables. Subsequently, we
describe two new generative image models that exploit different image transformations
as latent variables: a quantized grayscale view of the image or a multi-resolution
image pyramid. The proposed models tackle two known shortcomings of existing PixelCNN
models: 1) their tendency to focus on low-level image details, while largely ignoring
high-level image information, such as object shapes, and 2) their computationally
costly procedure for image sampling. We experimentally demonstrate benefits of
our LatentPixelCNN models, in particular showing that they produce much more realistically
looking image samples than previous state-of-the-art probabilistic models. '
acknowledgement: We thank Tim Salimans for spotting a mistake in our preliminary arXiv
manuscript. This work was funded by the European Research Council under the European
Unions Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no 308036.
article_processing_charge: No
author:
- first_name: Alexander
full_name: Kolesnikov, Alexander
id: 2D157DB6-F248-11E8-B48F-1D18A9856A87
last_name: Kolesnikov
- first_name: Christoph
full_name: Lampert, Christoph
id: 40C20FD2-F248-11E8-B48F-1D18A9856A87
last_name: Lampert
orcid: 0000-0001-8622-7887
citation:
ama: 'Kolesnikov A, Lampert C. PixelCNN models with auxiliary variables for natural
image modeling. In: 34th International Conference on Machine Learning.
Vol 70. JMLR; 2017:1905-1914.'
apa: 'Kolesnikov, A., & Lampert, C. (2017). PixelCNN models with auxiliary variables
for natural image modeling. In 34th International Conference on Machine Learning
(Vol. 70, pp. 1905–1914). Sydney, Australia: JMLR.'
chicago: Kolesnikov, Alexander, and Christoph Lampert. “PixelCNN Models with Auxiliary
Variables for Natural Image Modeling.” In 34th International Conference on
Machine Learning, 70:1905–14. JMLR, 2017.
ieee: A. Kolesnikov and C. Lampert, “PixelCNN models with auxiliary variables for
natural image modeling,” in 34th International Conference on Machine Learning,
Sydney, Australia, 2017, vol. 70, pp. 1905–1914.
ista: 'Kolesnikov A, Lampert C. 2017. PixelCNN models with auxiliary variables for
natural image modeling. 34th International Conference on Machine Learning. ICML:
International Conference on Machine Learning vol. 70, 1905–1914.'
mla: Kolesnikov, Alexander, and Christoph Lampert. “PixelCNN Models with Auxiliary
Variables for Natural Image Modeling.” 34th International Conference on Machine
Learning, vol. 70, JMLR, 2017, pp. 1905–14.
short: A. Kolesnikov, C. Lampert, in:, 34th International Conference on Machine
Learning, JMLR, 2017, pp. 1905–1914.
conference:
end_date: 2017-08-11
location: Sydney, Australia
name: 'ICML: International Conference on Machine Learning'
start_date: 2017-08-06
date_created: 2018-12-11T11:49:37Z
date_published: 2017-08-01T00:00:00Z
date_updated: 2023-09-22T09:50:41Z
day: '01'
department:
- _id: ChLa
ec_funded: 1
external_id:
arxiv:
- '1612.08185'
isi:
- '000683309501102'
has_accepted_license: '1'
intvolume: ' 70'
isi: 1
language:
- iso: eng
main_file_link:
- open_access: '1'
url: https://arxiv.org/abs/1612.08185
month: '08'
oa: 1
oa_version: Submitted Version
page: 1905 - 1914
project:
- _id: 2532554C-B435-11E9-9278-68D0E5697425
call_identifier: FP7
grant_number: '308036'
name: Lifelong Learning of Visual Scene Understanding
publication: 34th International Conference on Machine Learning
publication_identifier:
isbn:
- 978-151085514-4
publication_status: published
publisher: JMLR
publist_id: '6398'
quality_controlled: '1'
scopus_import: '1'
status: public
title: PixelCNN models with auxiliary variables for natural image modeling
type: conference
user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1
volume: 70
year: '2017'
...
---
_id: '432'
abstract:
- lang: eng
text: 'Recently there has been significant interest in training machine-learning
models at low precision: by reducing precision, one can reduce computation and
communication by one order of magnitude. We examine training at reduced precision,
both from a theoretical and practical perspective, and ask: is it possible to
train models at end-to-end low precision with provable guarantees? Can this lead
to consistent order-of-magnitude speedups? We mainly focus on linear models, and
the answer is yes for linear models. We develop a simple framework called ZipML
based on one simple but novel strategy called double sampling. Our ZipML framework
is able to execute training at low precision with no bias, guaranteeing convergence,
whereas naive quanti- zation would introduce significant bias. We val- idate our
framework across a range of applica- tions, and show that it enables an FPGA proto-
type that is up to 6.5 × faster than an implemen- tation using full 32-bit precision.
We further de- velop a variance-optimal stochastic quantization strategy and show
that it can make a significant difference in a variety of settings. When applied
to linear models together with double sampling, we save up to another 1.7 × in
data movement compared with uniform quantization. When training deep networks
with quantized models, we achieve higher accuracy than the state-of-the- art XNOR-Net. '
alternative_title:
- PMLR Press
article_processing_charge: No
author:
- first_name: Hantian
full_name: Zhang, Hantian
last_name: Zhang
- first_name: Jerry
full_name: Li, Jerry
last_name: Li
- first_name: Kaan
full_name: Kara, Kaan
last_name: Kara
- first_name: Dan-Adrian
full_name: Alistarh, Dan-Adrian
id: 4A899BFC-F248-11E8-B48F-1D18A9856A87
last_name: Alistarh
orcid: 0000-0003-3650-940X
- first_name: Ji
full_name: Liu, Ji
last_name: Liu
- first_name: Ce
full_name: Zhang, Ce
last_name: Zhang
citation:
ama: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. ZipML: Training linear
models with end-to-end low precision, and a little bit of deep learning. In: Proceedings
of Machine Learning Research. Vol 70. ML Research Press; 2017:4035-4043.'
apa: 'Zhang, H., Li, J., Kara, K., Alistarh, D.-A., Liu, J., & Zhang, C. (2017).
ZipML: Training linear models with end-to-end low precision, and a little bit
of deep learning. In Proceedings of Machine Learning Research (Vol. 70,
pp. 4035–4043). Sydney, Australia: ML Research Press.'
chicago: 'Zhang, Hantian, Jerry Li, Kaan Kara, Dan-Adrian Alistarh, Ji Liu, and
Ce Zhang. “ZipML: Training Linear Models with End-to-End Low Precision, and a
Little Bit of Deep Learning.” In Proceedings of Machine Learning Research,
70:4035–43. ML Research Press, 2017.'
ieee: 'H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, and C. Zhang, “ZipML: Training
linear models with end-to-end low precision, and a little bit of deep learning,”
in Proceedings of Machine Learning Research, Sydney, Australia, 2017, vol.
70, pp. 4035–4043.'
ista: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. 2017. ZipML: Training
linear models with end-to-end low precision, and a little bit of deep learning.
Proceedings of Machine Learning Research. ICML: International Conference on
Machine Learning, PMLR Press, vol. 70, 4035–4043.'
mla: 'Zhang, Hantian, et al. “ZipML: Training Linear Models with End-to-End Low
Precision, and a Little Bit of Deep Learning.” Proceedings of Machine Learning
Research, vol. 70, ML Research Press, 2017, pp. 4035–43.'
short: H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, C. Zhang, in:, Proceedings
of Machine Learning Research, ML Research Press, 2017, pp. 4035–4043.
conference:
end_date: 2017-08-11
location: Sydney, Australia
name: 'ICML: International Conference on Machine Learning'
start_date: 2017-08-06
date_created: 2018-12-11T11:46:26Z
date_published: 2017-01-01T00:00:00Z
date_updated: 2023-10-17T12:31:15Z
day: '01'
ddc:
- '000'
department:
- _id: DaAl
file:
- access_level: open_access
checksum: 86156ba7f4318e47cef3eb9092593c10
content_type: application/pdf
creator: dernst
date_created: 2019-01-22T08:23:58Z
date_updated: 2020-07-14T12:46:26Z
file_id: '5869'
file_name: 2017_ICML_Zhang.pdf
file_size: 849345
relation: main_file
file_date_updated: 2020-07-14T12:46:26Z
has_accepted_license: '1'
language:
- iso: eng
month: '01'
oa: 1
oa_version: Submitted Version
page: 4035 - 4043
publication: Proceedings of Machine Learning Research
publication_identifier:
isbn:
- 978-151085514-4
publication_status: published
publisher: ML Research Press
publist_id: '7391'
quality_controlled: '1'
scopus_import: '1'
status: public
title: 'ZipML: Training linear models with end-to-end low precision, and a little
bit of deep learning'
type: conference
user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87
volume: ' 70'
year: '2017'
...