--- _id: '1000' abstract: - lang: eng text: 'We study probabilistic models of natural images and extend the autoregressive family of PixelCNN models by incorporating latent variables. Subsequently, we describe two new generative image models that exploit different image transformations as latent variables: a quantized grayscale view of the image or a multi-resolution image pyramid. The proposed models tackle two known shortcomings of existing PixelCNN models: 1) their tendency to focus on low-level image details, while largely ignoring high-level image information, such as object shapes, and 2) their computationally costly procedure for image sampling. We experimentally demonstrate benefits of our LatentPixelCNN models, in particular showing that they produce much more realistically looking image samples than previous state-of-the-art probabilistic models. ' acknowledgement: We thank Tim Salimans for spotting a mistake in our preliminary arXiv manuscript. This work was funded by the European Research Council under the European Unions Seventh Framework Programme (FP7/2007-2013)/ERC grant agreement no 308036. article_processing_charge: No author: - first_name: Alexander full_name: Kolesnikov, Alexander id: 2D157DB6-F248-11E8-B48F-1D18A9856A87 last_name: Kolesnikov - first_name: Christoph full_name: Lampert, Christoph id: 40C20FD2-F248-11E8-B48F-1D18A9856A87 last_name: Lampert orcid: 0000-0001-8622-7887 citation: ama: 'Kolesnikov A, Lampert C. PixelCNN models with auxiliary variables for natural image modeling. In: 34th International Conference on Machine Learning. Vol 70. JMLR; 2017:1905-1914.' apa: 'Kolesnikov, A., & Lampert, C. (2017). PixelCNN models with auxiliary variables for natural image modeling. In 34th International Conference on Machine Learning (Vol. 70, pp. 1905–1914). Sydney, Australia: JMLR.' chicago: Kolesnikov, Alexander, and Christoph Lampert. “PixelCNN Models with Auxiliary Variables for Natural Image Modeling.” In 34th International Conference on Machine Learning, 70:1905–14. JMLR, 2017. ieee: A. Kolesnikov and C. Lampert, “PixelCNN models with auxiliary variables for natural image modeling,” in 34th International Conference on Machine Learning, Sydney, Australia, 2017, vol. 70, pp. 1905–1914. ista: 'Kolesnikov A, Lampert C. 2017. PixelCNN models with auxiliary variables for natural image modeling. 34th International Conference on Machine Learning. ICML: International Conference on Machine Learning vol. 70, 1905–1914.' mla: Kolesnikov, Alexander, and Christoph Lampert. “PixelCNN Models with Auxiliary Variables for Natural Image Modeling.” 34th International Conference on Machine Learning, vol. 70, JMLR, 2017, pp. 1905–14. short: A. Kolesnikov, C. Lampert, in:, 34th International Conference on Machine Learning, JMLR, 2017, pp. 1905–1914. conference: end_date: 2017-08-11 location: Sydney, Australia name: 'ICML: International Conference on Machine Learning' start_date: 2017-08-06 date_created: 2018-12-11T11:49:37Z date_published: 2017-08-01T00:00:00Z date_updated: 2023-09-22T09:50:41Z day: '01' department: - _id: ChLa ec_funded: 1 external_id: arxiv: - '1612.08185' isi: - '000683309501102' has_accepted_license: '1' intvolume: ' 70' isi: 1 language: - iso: eng main_file_link: - open_access: '1' url: https://arxiv.org/abs/1612.08185 month: '08' oa: 1 oa_version: Submitted Version page: 1905 - 1914 project: - _id: 2532554C-B435-11E9-9278-68D0E5697425 call_identifier: FP7 grant_number: '308036' name: Lifelong Learning of Visual Scene Understanding publication: 34th International Conference on Machine Learning publication_identifier: isbn: - 978-151085514-4 publication_status: published publisher: JMLR publist_id: '6398' quality_controlled: '1' scopus_import: '1' status: public title: PixelCNN models with auxiliary variables for natural image modeling type: conference user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1 volume: 70 year: '2017' ... --- _id: '432' abstract: - lang: eng text: 'Recently there has been significant interest in training machine-learning models at low precision: by reducing precision, one can reduce computation and communication by one order of magnitude. We examine training at reduced precision, both from a theoretical and practical perspective, and ask: is it possible to train models at end-to-end low precision with provable guarantees? Can this lead to consistent order-of-magnitude speedups? We mainly focus on linear models, and the answer is yes for linear models. We develop a simple framework called ZipML based on one simple but novel strategy called double sampling. Our ZipML framework is able to execute training at low precision with no bias, guaranteeing convergence, whereas naive quanti- zation would introduce significant bias. We val- idate our framework across a range of applica- tions, and show that it enables an FPGA proto- type that is up to 6.5 × faster than an implemen- tation using full 32-bit precision. We further de- velop a variance-optimal stochastic quantization strategy and show that it can make a significant difference in a variety of settings. When applied to linear models together with double sampling, we save up to another 1.7 × in data movement compared with uniform quantization. When training deep networks with quantized models, we achieve higher accuracy than the state-of-the- art XNOR-Net. ' alternative_title: - PMLR Press article_processing_charge: No author: - first_name: Hantian full_name: Zhang, Hantian last_name: Zhang - first_name: Jerry full_name: Li, Jerry last_name: Li - first_name: Kaan full_name: Kara, Kaan last_name: Kara - first_name: Dan-Adrian full_name: Alistarh, Dan-Adrian id: 4A899BFC-F248-11E8-B48F-1D18A9856A87 last_name: Alistarh orcid: 0000-0003-3650-940X - first_name: Ji full_name: Liu, Ji last_name: Liu - first_name: Ce full_name: Zhang, Ce last_name: Zhang citation: ama: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. ZipML: Training linear models with end-to-end low precision, and a little bit of deep learning. In: Proceedings of Machine Learning Research. Vol 70. ML Research Press; 2017:4035-4043.' apa: 'Zhang, H., Li, J., Kara, K., Alistarh, D.-A., Liu, J., & Zhang, C. (2017). ZipML: Training linear models with end-to-end low precision, and a little bit of deep learning. In Proceedings of Machine Learning Research (Vol. 70, pp. 4035–4043). Sydney, Australia: ML Research Press.' chicago: 'Zhang, Hantian, Jerry Li, Kaan Kara, Dan-Adrian Alistarh, Ji Liu, and Ce Zhang. “ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning.” In Proceedings of Machine Learning Research, 70:4035–43. ML Research Press, 2017.' ieee: 'H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, and C. Zhang, “ZipML: Training linear models with end-to-end low precision, and a little bit of deep learning,” in Proceedings of Machine Learning Research, Sydney, Australia, 2017, vol. 70, pp. 4035–4043.' ista: 'Zhang H, Li J, Kara K, Alistarh D-A, Liu J, Zhang C. 2017. ZipML: Training linear models with end-to-end low precision, and a little bit of deep learning. Proceedings of Machine Learning Research. ICML: International  Conference  on  Machine Learning, PMLR Press, vol. 70, 4035–4043.' mla: 'Zhang, Hantian, et al. “ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning.” Proceedings of Machine Learning Research, vol. 70, ML Research Press, 2017, pp. 4035–43.' short: H. Zhang, J. Li, K. Kara, D.-A. Alistarh, J. Liu, C. Zhang, in:, Proceedings of Machine Learning Research, ML Research Press, 2017, pp. 4035–4043. conference: end_date: 2017-08-11 location: Sydney, Australia name: 'ICML: International Conference on Machine Learning' start_date: 2017-08-06 date_created: 2018-12-11T11:46:26Z date_published: 2017-01-01T00:00:00Z date_updated: 2023-10-17T12:31:15Z day: '01' ddc: - '000' department: - _id: DaAl file: - access_level: open_access checksum: 86156ba7f4318e47cef3eb9092593c10 content_type: application/pdf creator: dernst date_created: 2019-01-22T08:23:58Z date_updated: 2020-07-14T12:46:26Z file_id: '5869' file_name: 2017_ICML_Zhang.pdf file_size: 849345 relation: main_file file_date_updated: 2020-07-14T12:46:26Z has_accepted_license: '1' language: - iso: eng month: '01' oa: 1 oa_version: Submitted Version page: 4035 - 4043 publication: Proceedings of Machine Learning Research publication_identifier: isbn: - 978-151085514-4 publication_status: published publisher: ML Research Press publist_id: '7391' quality_controlled: '1' scopus_import: '1' status: public title: 'ZipML: Training linear models with end-to-end low precision, and a little bit of deep learning' type: conference user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: ' 70' year: '2017' ...