--- _id: '7639' abstract: - lang: eng text: Deep neural networks (DNNs) have become increasingly important due to their excellent empirical performance on a wide range of problems. However, regularization is generally achieved by indirect means, largely due to the complex set of functions defined by a network and the difficulty in measuring function complexity. There exists no method in the literature for additive regularization based on a norm of the function, as is classically considered in statistical learning theory. In this work, we study the tractability of function norms for deep neural networks with ReLU activations. We provide, to the best of our knowledge, the first proof in the literature of the NP-hardness of computing function norms of DNNs of 3 or more layers. We also highlight a fundamental difference between shallow and deep networks. In the light on these results, we propose a new regularization strategy based on approximate function norms, and show its efficiency on a segmentation task with a DNN. article_number: 748-752 article_processing_charge: No author: - first_name: Amal full_name: Rannen-Triki, Amal last_name: Rannen-Triki - first_name: Maxim full_name: Berman, Maxim last_name: Berman - first_name: Vladimir full_name: Kolmogorov, Vladimir id: 3D50B0BA-F248-11E8-B48F-1D18A9856A87 last_name: Kolmogorov - first_name: Matthew B. full_name: Blaschko, Matthew B. last_name: Blaschko citation: ama: 'Rannen-Triki A, Berman M, Kolmogorov V, Blaschko MB. Function norms for neural networks. In: Proceedings of the 2019 International Conference on Computer Vision Workshop. IEEE; 2019. doi:10.1109/ICCVW.2019.00097' apa: 'Rannen-Triki, A., Berman, M., Kolmogorov, V., & Blaschko, M. B. (2019). Function norms for neural networks. In Proceedings of the 2019 International Conference on Computer Vision Workshop. Seoul, South Korea: IEEE. https://doi.org/10.1109/ICCVW.2019.00097' chicago: Rannen-Triki, Amal, Maxim Berman, Vladimir Kolmogorov, and Matthew B. Blaschko. “Function Norms for Neural Networks.” In Proceedings of the 2019 International Conference on Computer Vision Workshop. IEEE, 2019. https://doi.org/10.1109/ICCVW.2019.00097. ieee: A. Rannen-Triki, M. Berman, V. Kolmogorov, and M. B. Blaschko, “Function norms for neural networks,” in Proceedings of the 2019 International Conference on Computer Vision Workshop, Seoul, South Korea, 2019. ista: 'Rannen-Triki A, Berman M, Kolmogorov V, Blaschko MB. 2019. Function norms for neural networks. Proceedings of the 2019 International Conference on Computer Vision Workshop. ICCVW: International Conference on Computer Vision Workshop, 748–752.' mla: Rannen-Triki, Amal, et al. “Function Norms for Neural Networks.” Proceedings of the 2019 International Conference on Computer Vision Workshop, 748–752, IEEE, 2019, doi:10.1109/ICCVW.2019.00097. short: A. Rannen-Triki, M. Berman, V. Kolmogorov, M.B. Blaschko, in:, Proceedings of the 2019 International Conference on Computer Vision Workshop, IEEE, 2019. conference: end_date: 2019-10-28 location: Seoul, South Korea name: 'ICCVW: International Conference on Computer Vision Workshop' start_date: 2019-10-27 date_created: 2020-04-05T22:00:50Z date_published: 2019-10-01T00:00:00Z date_updated: 2023-09-08T11:19:12Z day: '01' department: - _id: VlKo doi: 10.1109/ICCVW.2019.00097 external_id: isi: - '000554591600090' isi: 1 language: - iso: eng month: '10' oa_version: None publication: Proceedings of the 2019 International Conference on Computer Vision Workshop publication_identifier: isbn: - '9781728150239' publication_status: published publisher: IEEE quality_controlled: '1' scopus_import: '1' status: public title: Function norms for neural networks type: conference user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1 year: '2019' ... --- _id: '703' abstract: - lang: eng text: We consider the NP-hard problem of MAP-inference for undirected discrete graphical models. We propose a polynomial time and practically efficient algorithm for finding a part of its optimal solution. Specifically, our algorithm marks some labels of the considered graphical model either as (i) optimal, meaning that they belong to all optimal solutions of the inference problem; (ii) non-optimal if they provably do not belong to any solution. With access to an exact solver of a linear programming relaxation to the MAP-inference problem, our algorithm marks the maximal possible (in a specified sense) number of labels. We also present a version of the algorithm, which has access to a suboptimal dual solver only and still can ensure the (non-)optimality for the marked labels, although the overall number of the marked labels may decrease. We propose an efficient implementation, which runs in time comparable to a single run of a suboptimal dual solver. Our method is well-scalable and shows state-of-the-art results on computational benchmarks from machine learning and computer vision. author: - first_name: Alexander full_name: Shekhovtsov, Alexander last_name: Shekhovtsov - first_name: Paul full_name: Swoboda, Paul id: 446560C6-F248-11E8-B48F-1D18A9856A87 last_name: Swoboda - first_name: Bogdan full_name: Savchynskyy, Bogdan last_name: Savchynskyy citation: ama: Shekhovtsov A, Swoboda P, Savchynskyy B. Maximum persistency via iterative relaxed inference with graphical models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2018;40(7):1668-1682. doi:10.1109/TPAMI.2017.2730884 apa: Shekhovtsov, A., Swoboda, P., & Savchynskyy, B. (2018). Maximum persistency via iterative relaxed inference with graphical models. IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE. https://doi.org/10.1109/TPAMI.2017.2730884 chicago: Shekhovtsov, Alexander, Paul Swoboda, and Bogdan Savchynskyy. “Maximum Persistency via Iterative Relaxed Inference with Graphical Models.” IEEE Transactions on Pattern Analysis and Machine Intelligence. IEEE, 2018. https://doi.org/10.1109/TPAMI.2017.2730884. ieee: A. Shekhovtsov, P. Swoboda, and B. Savchynskyy, “Maximum persistency via iterative relaxed inference with graphical models,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 7. IEEE, pp. 1668–1682, 2018. ista: Shekhovtsov A, Swoboda P, Savchynskyy B. 2018. Maximum persistency via iterative relaxed inference with graphical models. IEEE Transactions on Pattern Analysis and Machine Intelligence. 40(7), 1668–1682. mla: Shekhovtsov, Alexander, et al. “Maximum Persistency via Iterative Relaxed Inference with Graphical Models.” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, no. 7, IEEE, 2018, pp. 1668–82, doi:10.1109/TPAMI.2017.2730884. short: A. Shekhovtsov, P. Swoboda, B. Savchynskyy, IEEE Transactions on Pattern Analysis and Machine Intelligence 40 (2018) 1668–1682. date_created: 2018-12-11T11:48:01Z date_published: 2018-07-01T00:00:00Z date_updated: 2021-01-12T08:11:32Z day: '01' department: - _id: VlKo doi: 10.1109/TPAMI.2017.2730884 external_id: arxiv: - '1508.07902' intvolume: ' 40' issue: '7' language: - iso: eng main_file_link: - open_access: '1' url: https://arxiv.org/abs/1508.07902 month: '07' oa: 1 oa_version: Preprint page: 1668-1682 publication: IEEE Transactions on Pattern Analysis and Machine Intelligence publication_identifier: issn: - '01628828' publication_status: published publisher: IEEE publist_id: '6992' quality_controlled: '1' scopus_import: 1 status: public title: Maximum persistency via iterative relaxed inference with graphical models type: journal_article user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 40 year: '2018' ... --- _id: '10864' abstract: - lang: eng text: We prove that every congruence distributive variety has directed Jónsson terms, and every congruence modular variety has directed Gumm terms. The directed terms we construct witness every case of absorption witnessed by the original Jónsson or Gumm terms. This result is equivalent to a pair of claims about absorption for admissible preorders in congruence distributive and congruence modular varieties, respectively. For finite algebras, these absorption theorems have already seen significant applications, but until now, it was not clear if the theorems hold for general algebras as well. Our method also yields a novel proof of a result by P. Lipparini about the existence of a chain of terms (which we call Pixley terms) in varieties that are at the same time congruence distributive and k-permutable for some k. acknowledgement: The second author was supported by National Science Center grant DEC-2011-/01/B/ST6/01006. article_processing_charge: No author: - first_name: Alexandr full_name: Kazda, Alexandr id: 3B32BAA8-F248-11E8-B48F-1D18A9856A87 last_name: Kazda - first_name: Marcin full_name: Kozik, Marcin last_name: Kozik - first_name: Ralph full_name: McKenzie, Ralph last_name: McKenzie - first_name: Matthew full_name: Moore, Matthew last_name: Moore citation: ama: 'Kazda A, Kozik M, McKenzie R, Moore M. Absorption and directed Jónsson terms. In: Czelakowski J, ed. Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science. Vol 16. OCTR. Cham: Springer Nature; 2018:203-220. doi:10.1007/978-3-319-74772-9_7' apa: 'Kazda, A., Kozik, M., McKenzie, R., & Moore, M. (2018). Absorption and directed Jónsson terms. In J. Czelakowski (Ed.), Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science (Vol. 16, pp. 203–220). Cham: Springer Nature. https://doi.org/10.1007/978-3-319-74772-9_7' chicago: 'Kazda, Alexandr, Marcin Kozik, Ralph McKenzie, and Matthew Moore. “Absorption and Directed Jónsson Terms.” In Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science, edited by J Czelakowski, 16:203–20. OCTR. Cham: Springer Nature, 2018. https://doi.org/10.1007/978-3-319-74772-9_7.' ieee: 'A. Kazda, M. Kozik, R. McKenzie, and M. Moore, “Absorption and directed Jónsson terms,” in Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science, vol. 16, J. Czelakowski, Ed. Cham: Springer Nature, 2018, pp. 203–220.' ista: 'Kazda A, Kozik M, McKenzie R, Moore M. 2018.Absorption and directed Jónsson terms. In: Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science. vol. 16, 203–220.' mla: Kazda, Alexandr, et al. “Absorption and Directed Jónsson Terms.” Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science, edited by J Czelakowski, vol. 16, Springer Nature, 2018, pp. 203–20, doi:10.1007/978-3-319-74772-9_7. short: A. Kazda, M. Kozik, R. McKenzie, M. Moore, in:, J. Czelakowski (Ed.), Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science, Springer Nature, Cham, 2018, pp. 203–220. date_created: 2022-03-18T10:30:32Z date_published: 2018-03-21T00:00:00Z date_updated: 2023-09-05T15:37:18Z day: '21' department: - _id: VlKo doi: 10.1007/978-3-319-74772-9_7 editor: - first_name: J full_name: Czelakowski, J last_name: Czelakowski external_id: arxiv: - '1502.01072' intvolume: ' 16' language: - iso: eng main_file_link: - open_access: '1' url: https://arxiv.org/abs/1502.01072 month: '03' oa: 1 oa_version: Preprint page: 203-220 place: Cham publication: Don Pigozzi on Abstract Algebraic Logic, Universal Algebra, and Computer Science publication_identifier: eisbn: - '9783319747729' eissn: - 2211-2766 isbn: - '9783319747712' issn: - 2211-2758 publication_status: published publisher: Springer Nature quality_controlled: '1' scopus_import: '1' series_title: OCTR status: public title: Absorption and directed Jónsson terms type: book_chapter user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1 volume: 16 year: '2018' ... --- _id: '273' abstract: - lang: eng text: The accuracy of information retrieval systems is often measured using complex loss functions such as the average precision (AP) or the normalized discounted cumulative gain (NDCG). Given a set of positive and negative samples, the parameters of a retrieval system can be estimated by minimizing these loss functions. However, the non-differentiability and non-decomposability of these loss functions does not allow for simple gradient based optimization algorithms. This issue is generally circumvented by either optimizing a structured hinge-loss upper bound to the loss function or by using asymptotic methods like the direct-loss minimization framework. Yet, the high computational complexity of loss-augmented inference, which is necessary for both the frameworks, prohibits its use in large training data sets. To alleviate this deficiency, we present a novel quicksort flavored algorithm for a large class of non-decomposable loss functions. We provide a complete characterization of the loss functions that are amenable to our algorithm, and show that it includes both AP and NDCG based loss functions. Furthermore, we prove that no comparison based algorithm can improve upon the computational complexity of our approach asymptotically. We demonstrate the effectiveness of our approach in the context of optimizing the structured hinge loss upper bound of AP and NDCG loss for learning models for a variety of vision tasks. We show that our approach provides significantly better results than simpler decomposable loss functions, while requiring a comparable training time. article_processing_charge: No author: - first_name: Pritish full_name: Mohapatra, Pritish last_name: Mohapatra - first_name: Michal full_name: Rolinek, Michal id: 3CB3BC06-F248-11E8-B48F-1D18A9856A87 last_name: Rolinek - first_name: C V full_name: Jawahar, C V last_name: Jawahar - first_name: Vladimir full_name: Kolmogorov, Vladimir id: 3D50B0BA-F248-11E8-B48F-1D18A9856A87 last_name: Kolmogorov - first_name: M Pawan full_name: Kumar, M Pawan last_name: Kumar citation: ama: 'Mohapatra P, Rolinek M, Jawahar CV, Kolmogorov V, Kumar MP. Efficient optimization for rank-based loss functions. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE; 2018:3693-3701. doi:10.1109/cvpr.2018.00389' apa: 'Mohapatra, P., Rolinek, M., Jawahar, C. V., Kolmogorov, V., & Kumar, M. P. (2018). Efficient optimization for rank-based loss functions. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3693–3701). Salt Lake City, UT, USA: IEEE. https://doi.org/10.1109/cvpr.2018.00389' chicago: Mohapatra, Pritish, Michal Rolinek, C V Jawahar, Vladimir Kolmogorov, and M Pawan Kumar. “Efficient Optimization for Rank-Based Loss Functions.” In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3693–3701. IEEE, 2018. https://doi.org/10.1109/cvpr.2018.00389. ieee: P. Mohapatra, M. Rolinek, C. V. Jawahar, V. Kolmogorov, and M. P. Kumar, “Efficient optimization for rank-based loss functions,” in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 3693–3701. ista: 'Mohapatra P, Rolinek M, Jawahar CV, Kolmogorov V, Kumar MP. 2018. Efficient optimization for rank-based loss functions. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. CVPR: Conference on Computer Vision and Pattern Recognition, 3693–3701.' mla: Mohapatra, Pritish, et al. “Efficient Optimization for Rank-Based Loss Functions.” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2018, pp. 3693–701, doi:10.1109/cvpr.2018.00389. short: P. Mohapatra, M. Rolinek, C.V. Jawahar, V. Kolmogorov, M.P. Kumar, in:, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, IEEE, 2018, pp. 3693–3701. conference: end_date: 2018-06-22 location: Salt Lake City, UT, USA name: 'CVPR: Conference on Computer Vision and Pattern Recognition' start_date: 2018-06-18 date_created: 2018-12-11T11:45:33Z date_published: 2018-06-28T00:00:00Z date_updated: 2023-09-11T13:24:43Z day: '28' department: - _id: VlKo doi: 10.1109/cvpr.2018.00389 ec_funded: 1 external_id: arxiv: - '1604.08269' isi: - '000457843603087' isi: 1 language: - iso: eng main_file_link: - open_access: '1' url: https://arxiv.org/abs/1604.08269 month: '06' oa: 1 oa_version: Preprint page: 3693-3701 project: - _id: 25FBA906-B435-11E9-9278-68D0E5697425 call_identifier: FP7 grant_number: '616160' name: 'Discrete Optimization in Computer Vision: Theory and Practice' publication: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition publication_identifier: isbn: - '9781538664209' publication_status: published publisher: IEEE quality_controlled: '1' scopus_import: '1' status: public title: Efficient optimization for rank-based loss functions type: conference user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1 year: '2018' ... --- _id: '193' abstract: - lang: eng text: 'We show attacks on five data-independent memory-hard functions (iMHF) that were submitted to the password hashing competition (PHC). Informally, an MHF is a function which cannot be evaluated on dedicated hardware, like ASICs, at significantly lower hardware and/or energy cost than evaluating a single instance on a standard single-core architecture. Data-independent means the memory access pattern of the function is independent of the input; this makes iMHFs harder to construct than data-dependent ones, but the latter can be attacked by various side-channel attacks. Following [Alwen-Blocki''16], we capture the evaluation of an iMHF as a directed acyclic graph (DAG). The cumulative parallel pebbling complexity of this DAG is a measure for the hardware cost of evaluating the iMHF on an ASIC. Ideally, one would like the complexity of a DAG underlying an iMHF to be as close to quadratic in the number of nodes of the graph as possible. Instead, we show that (the DAGs underlying) the following iMHFs are far from this bound: Rig.v2, TwoCats and Gambit each having an exponent no more than 1.75. Moreover, we show that the complexity of the iMHF modes of the PHC finalists Pomelo and Lyra2 have exponents at most 1.83 and 1.67 respectively. To show this we investigate a combinatorial property of each underlying DAG (called its depth-robustness. By establishing upper bounds on this property we are then able to apply the general technique of [Alwen-Block''16] for analyzing the hardware costs of an iMHF.' acknowledgement: Leonid Reyzin was supported in part by IST Austria and by US NSF grants 1012910, 1012798, and 1422965; this research was performed while he was visiting IST Austria. article_processing_charge: No author: - first_name: Joel F full_name: Alwen, Joel F id: 2A8DFA8C-F248-11E8-B48F-1D18A9856A87 last_name: Alwen - first_name: Peter full_name: Gazi, Peter last_name: Gazi - first_name: Chethan full_name: Kamath Hosdurg, Chethan id: 4BD3F30E-F248-11E8-B48F-1D18A9856A87 last_name: Kamath Hosdurg - first_name: Karen full_name: Klein, Karen id: 3E83A2F8-F248-11E8-B48F-1D18A9856A87 last_name: Klein - first_name: Georg F full_name: Osang, Georg F id: 464B40D6-F248-11E8-B48F-1D18A9856A87 last_name: Osang orcid: 0000-0002-8882-5116 - first_name: Krzysztof Z full_name: Pietrzak, Krzysztof Z id: 3E04A7AA-F248-11E8-B48F-1D18A9856A87 last_name: Pietrzak orcid: 0000-0002-9139-1654 - first_name: Lenoid full_name: Reyzin, Lenoid last_name: Reyzin - first_name: Michal full_name: Rolinek, Michal id: 3CB3BC06-F248-11E8-B48F-1D18A9856A87 last_name: Rolinek - first_name: Michal full_name: Rybar, Michal id: 2B3E3DE8-F248-11E8-B48F-1D18A9856A87 last_name: Rybar citation: ama: 'Alwen JF, Gazi P, Kamath Hosdurg C, et al. On the memory hardness of data independent password hashing functions. In: Proceedings of the 2018 on Asia Conference on Computer and Communication Security. ACM; 2018:51-65. doi:10.1145/3196494.3196534' apa: 'Alwen, J. F., Gazi, P., Kamath Hosdurg, C., Klein, K., Osang, G. F., Pietrzak, K. Z., … Rybar, M. (2018). On the memory hardness of data independent password hashing functions. In Proceedings of the 2018 on Asia Conference on Computer and Communication Security (pp. 51–65). Incheon, Republic of Korea: ACM. https://doi.org/10.1145/3196494.3196534' chicago: Alwen, Joel F, Peter Gazi, Chethan Kamath Hosdurg, Karen Klein, Georg F Osang, Krzysztof Z Pietrzak, Lenoid Reyzin, Michal Rolinek, and Michal Rybar. “On the Memory Hardness of Data Independent Password Hashing Functions.” In Proceedings of the 2018 on Asia Conference on Computer and Communication Security, 51–65. ACM, 2018. https://doi.org/10.1145/3196494.3196534. ieee: J. F. Alwen et al., “On the memory hardness of data independent password hashing functions,” in Proceedings of the 2018 on Asia Conference on Computer and Communication Security, Incheon, Republic of Korea, 2018, pp. 51–65. ista: 'Alwen JF, Gazi P, Kamath Hosdurg C, Klein K, Osang GF, Pietrzak KZ, Reyzin L, Rolinek M, Rybar M. 2018. On the memory hardness of data independent password hashing functions. Proceedings of the 2018 on Asia Conference on Computer and Communication Security. ASIACCS: Asia Conference on Computer and Communications Security , 51–65.' mla: Alwen, Joel F., et al. “On the Memory Hardness of Data Independent Password Hashing Functions.” Proceedings of the 2018 on Asia Conference on Computer and Communication Security, ACM, 2018, pp. 51–65, doi:10.1145/3196494.3196534. short: J.F. Alwen, P. Gazi, C. Kamath Hosdurg, K. Klein, G.F. Osang, K.Z. Pietrzak, L. Reyzin, M. Rolinek, M. Rybar, in:, Proceedings of the 2018 on Asia Conference on Computer and Communication Security, ACM, 2018, pp. 51–65. conference: end_date: 2018-06-08 location: Incheon, Republic of Korea name: 'ASIACCS: Asia Conference on Computer and Communications Security ' start_date: 2018-06-04 date_created: 2018-12-11T11:45:07Z date_published: 2018-06-01T00:00:00Z date_updated: 2023-09-13T09:13:12Z day: '01' department: - _id: KrPi - _id: HeEd - _id: VlKo doi: 10.1145/3196494.3196534 ec_funded: 1 external_id: isi: - '000516620100005' isi: 1 language: - iso: eng main_file_link: - open_access: '1' url: https://eprint.iacr.org/2016/783 month: '06' oa: 1 oa_version: Submitted Version page: 51 - 65 project: - _id: 25FBA906-B435-11E9-9278-68D0E5697425 call_identifier: FP7 grant_number: '616160' name: 'Discrete Optimization in Computer Vision: Theory and Practice' - _id: 258AA5B2-B435-11E9-9278-68D0E5697425 call_identifier: H2020 grant_number: '682815' name: Teaching Old Crypto New Tricks publication: Proceedings of the 2018 on Asia Conference on Computer and Communication Security publication_status: published publisher: ACM publist_id: '7723' quality_controlled: '1' scopus_import: '1' status: public title: On the memory hardness of data independent password hashing functions type: conference user_id: c635000d-4b10-11ee-a964-aac5a93f6ac1 year: '2018' ...