--- _id: '7130' abstract: - lang: eng text: "We show that statistical criticality, i.e. the occurrence of power law frequency distributions, arises in samples that are maximally informative about the underlying generating process. In order to reach this conclusion, we first identify the frequency with which different outcomes occur in a sample, as the variable carrying useful information on the generative process. The entropy of the frequency, that we call relevance, provides an upper bound to the number of informative bits. This differs from the entropy of the data, that we take as a measure of resolution. Samples that maximise relevance at a given resolution—that we call maximally informative samples—exhibit statistical criticality. In particular, Zipf's law arises at the optimal trade-off between resolution (i.e. compression) and relevance. As a byproduct, we derive a bound of the maximal number of parameters that can be estimated from a dataset, in the absence of prior knowledge on the generative model.\r\n\r\nFurthermore, we relate criticality to the statistical properties of the representation of the data generating process. We show that, as a consequence of the concentration property of the asymptotic equipartition property, representations that are maximally informative about the data generating process are characterised by an exponential distribution of energy levels. This arises from a principle of minimal entropy, that is conjugate of the maximum entropy principle in statistical mechanics. This explains why statistical criticality requires no parameter fine tuning in maximally informative samples." acknowledgement: We acknowledge interesting discussions with M Abbott, E Aurell, J Barbier, R Monasson, T Mora, I Nemenman, N Tishby and R Zecchina. This research was supported by the Kavli Foundation and the Centre of Excellence scheme of the Research Council of Norway (Centre for Neural Computation) (RJC and YR), by the Basic Science Research Program through the National Research Foundation of Korea (NRF), funded by the Ministry of Education (2016R1D1A1B03932264) (JJ), and, in part, by the ICTP through the OEA-AC-98 (JS). article_number: '063402' article_processing_charge: No article_type: original author: - first_name: Ryan J full_name: Cubero, Ryan J id: 850B2E12-9CD4-11E9-837F-E719E6697425 last_name: Cubero orcid: 0000-0003-0002-1867 - first_name: Junghyo full_name: Jo, Junghyo last_name: Jo - first_name: Matteo full_name: Marsili, Matteo last_name: Marsili - first_name: Yasser full_name: Roudi, Yasser last_name: Roudi - first_name: Juyong full_name: Song, Juyong last_name: Song citation: ama: 'Cubero RJ, Jo J, Marsili M, Roudi Y, Song J. Statistical criticality arises in most informative representations. Journal of Statistical Mechanics: Theory and Experiment. 2019;2019(6). doi:10.1088/1742-5468/ab16c8' apa: 'Cubero, R. J., Jo, J., Marsili, M., Roudi, Y., & Song, J. (2019). Statistical criticality arises in most informative representations. Journal of Statistical Mechanics: Theory and Experiment. IOP Publishing. https://doi.org/10.1088/1742-5468/ab16c8' chicago: 'Cubero, Ryan J, Junghyo Jo, Matteo Marsili, Yasser Roudi, and Juyong Song. “Statistical Criticality Arises in Most Informative Representations.” Journal of Statistical Mechanics: Theory and Experiment. IOP Publishing, 2019. https://doi.org/10.1088/1742-5468/ab16c8.' ieee: 'R. J. Cubero, J. Jo, M. Marsili, Y. Roudi, and J. Song, “Statistical criticality arises in most informative representations,” Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, no. 6. IOP Publishing, 2019.' ista: 'Cubero RJ, Jo J, Marsili M, Roudi Y, Song J. 2019. Statistical criticality arises in most informative representations. Journal of Statistical Mechanics: Theory and Experiment. 2019(6), 063402.' mla: 'Cubero, Ryan J., et al. “Statistical Criticality Arises in Most Informative Representations.” Journal of Statistical Mechanics: Theory and Experiment, vol. 2019, no. 6, 063402, IOP Publishing, 2019, doi:10.1088/1742-5468/ab16c8.' short: 'R.J. Cubero, J. Jo, M. Marsili, Y. Roudi, J. Song, Journal of Statistical Mechanics: Theory and Experiment 2019 (2019).' date_created: 2019-11-26T22:36:09Z date_published: 2019-06-17T00:00:00Z date_updated: 2021-01-12T08:11:57Z day: '17' doi: 10.1088/1742-5468/ab16c8 extern: '1' external_id: arxiv: - '1808.00249' intvolume: ' 2019' issue: '6' keyword: - optimization under uncertainty - source coding - large deviation language: - iso: eng main_file_link: - open_access: '1' url: https://arxiv.org/abs/1808.00249 month: '06' oa: 1 oa_version: Preprint publication: 'Journal of Statistical Mechanics: Theory and Experiment' publication_identifier: issn: - 1742-5468 publication_status: published publisher: IOP Publishing quality_controlled: '1' status: public title: Statistical criticality arises in most informative representations type: journal_article user_id: 2DF688A6-F248-11E8-B48F-1D18A9856A87 volume: 2019 year: '2019' ...