Minimum description length codes are critical

Cubero RJ, Marsili M, Roudi Y. 2018. Minimum description length codes are critical. Entropy. 20(10), 755.

Download
OA entropy-20-00755-v2.pdf 1.37 MB

Journal Article | Published | English
Author
Cubero, Ryan JohnIST Austria ; Marsili, Matteo; Roudi, Yasser
Abstract
In the Minimum Description Length (MDL) principle, learning from the data is equivalent to an optimal coding problem. We show that the codes that achieve optimal compression in MDL are critical in a very precise sense. First, when they are taken as generative models of samples, they generate samples with broad empirical distributions and with a high value of the relevance, defined as the entropy of the empirical frequencies. These results are derived for different statistical models (Dirichlet model, independent and pairwise dependent spin models, and restricted Boltzmann machines). Second, MDL codes sit precisely at a second order phase transition point where the symmetry between the sampled outcomes is spontaneously broken. The order parameter controlling the phase transition is the coding cost of the samples. The phase transition is a manifestation of the optimality of MDL codes, and it arises because codes that achieve a higher compression do not exist. These results suggest a clear interpretation of the widespread occurrence of statistical criticality as a characterization of samples which are maximally informative on the underlying generative process.
Publishing Year
Date Published
2018-10-01
Journal Title
Entropy
Volume
20
Issue
10
Article Number
755
ISSN
IST-REx-ID

Cite this

Cubero RJ, Marsili M, Roudi Y. Minimum description length codes are critical. Entropy. 2018;20(10). doi:10.3390/e20100755
Cubero, R. J., Marsili, M., & Roudi, Y. (2018). Minimum description length codes are critical. Entropy. MDPI. https://doi.org/10.3390/e20100755
Cubero, Ryan J, Matteo Marsili, and Yasser Roudi. “Minimum Description Length Codes Are Critical.” Entropy. MDPI, 2018. https://doi.org/10.3390/e20100755.
R. J. Cubero, M. Marsili, and Y. Roudi, “Minimum description length codes are critical,” Entropy, vol. 20, no. 10. MDPI, 2018.
Cubero RJ, Marsili M, Roudi Y. 2018. Minimum description length codes are critical. Entropy. 20(10), 755.
Cubero, Ryan J., et al. “Minimum Description Length Codes Are Critical.” Entropy, vol. 20, no. 10, 755, MDPI, 2018, doi:10.3390/e20100755.
All files available under the following license(s):
Creative Commons Attribution 4.0 International Public License (CC-BY 4.0):
Main File(s)
File Name
Access Level
OA Open Access
Date Uploaded
2019-11-26
MD5 Checksum
d642b7b661e1d5066b62e6ea9986b917


Export

Marked Publications

Open Data IST Research Explorer

Search this title in

Google Scholar