Summary

Information theory deals with encoding data in order to transmit it correctly and effectively. Statistics and machine learning deal with estimating models of data and predicting future observations. Is there any relationship between the two? It turns out, perhaps not surprisingly, that the most compact encoding of the data is by the probabilistic model that describes it best. In other words, there is a fundamental link between information and probability.

This course starts with the basic notions of information theory and explores its relationship to machine learning and statistics. The course will have a strong theoretical component, but will also focus on applications and computing. The topics to be covered are:

• Entropy, mutual information, Kullback-Leibler and Bregman divergences;
• Source-channel models; boosting and optimal betting strategy;
• Maximum likelihood and Bayesian inference;
• Channel capacity, rate distortion and information bottleneck method;
• Maximum entropy principle, information geometry and alternating algorithms;
• Large deviations, coding theory and approximate inference in graphical models.

Lectures

Time: Tuesday/Thursday 8:00 pm -9:15 pm; Location: Russ Center 155

Instructor

Shaojun Wang
428, Russ Engineering Center Building
shaojun.wang(at)wright.edu
(937) 775-5140
Office hours: Tuesday/Thursday 2:00PM-3:30PM

Textbook and Lecture Notes

Thomas M. Cover and Joy A. Thomas
Elements of Information Theory, 2 Edition
Wiley-Interscience, 2006.

I. Csiszar and P. Shields
Information Theory and Statistics: A Tutorial
Foundations and Trends in Communications and Information Theory, 1(4):417-528, 2004

David J. C. MacKay
Information Theory, Inference and Learning Algorithms
Cambridge University Press, 2003.

A. Montanari and R. Urbanke (2006).
Modern coding theory: the statistical mechanics and computer science points of view.
Complex Systems, LXXXV, Lecture Notes of the Les Houches Summer School.

Course Grades and Workload

Three Homeworks 60%
Projects 40% (Presentation: 20%; Report: 20%)

Prerequisite

A rudimentary knowledge of probability and statistics, for example, familiar with the materials in the standard textbook, A First Course in Probability by Sheldon Ross.