Relevant sparse codes with variational information bottleneck

M.J. Chalk, O. Marre, G. Tkacik, in:, Neural Information Processing Systems, 2016, pp. 1965–1973.

Conference Paper | Published | English
Department
Series Title
Advances in Neural Information Processing Systems
Abstract
In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximises information about a relevance variable, Y, while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximising a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelised versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y.
Publishing Year
Date Published
2016-12-01
Volume
29
Page
1965-1973
Conference
NIPS: Neural Information Processing Systems
Conference Location
Barcelona, Spain
Conference Date
2016-12-05 – 2016-12-10
IST-REx-ID

Cite this

Chalk MJ, Marre O, Tkacik G. Relevant sparse codes with variational information bottleneck. In: Vol 29. Neural Information Processing Systems; 2016:1965-1973.
Chalk, M. J., Marre, O., & Tkacik, G. (2016). Relevant sparse codes with variational information bottleneck (Vol. 29, pp. 1965–1973). Presented at the NIPS: Neural Information Processing Systems, Barcelona, Spain: Neural Information Processing Systems.
Chalk, Matthew J, Olivier Marre, and Gasper Tkacik. “Relevant Sparse Codes with Variational Information Bottleneck,” 29:1965–73. Neural Information Processing Systems, 2016.
M. J. Chalk, O. Marre, and G. Tkacik, “Relevant sparse codes with variational information bottleneck,” presented at the NIPS: Neural Information Processing Systems, Barcelona, Spain, 2016, vol. 29, pp. 1965–1973.
Chalk MJ, Marre O, Tkacik G. 2016. Relevant sparse codes with variational information bottleneck. NIPS: Neural Information Processing Systems, Advances in Neural Information Processing Systems, vol. 29. 1965–1973.
Chalk, Matthew J., et al. Relevant Sparse Codes with Variational Information Bottleneck. Vol. 29, Neural Information Processing Systems, 2016, pp. 1965–73.

Link(s) to Main File(s)
Access Level
OA Open Access

Export

Marked Publications

Open Data IST Research Explorer

Search this title in

Google Scholar