EE 740 Information Theory

EE 740: Information Theory

© Fred Garber, 2008. All Rights Reserved.
Note: Permission to copy and use all EE 740 course material for educational purposes is granted to any Wright State University student currently enrolled in EE 740.

Syllabus
Shannon's papers: A Mathematical Theory of Communication, Communication in the Presence of Noise
Discrete Entropy

Mathematica notebook addressing discrete entropy, conditional entropy, mutual information, and Kullback-leibler distance. Get/View PDF, Get Notebook.
MatLab: Here's a matlab file (by Phil Hanna) that performs the basic discrete entropy operations.
More MatLab: Here's a directory of .m files that perform the basic discrete entropy operations, plus Kullback-Leibler and the Kraft inequality.

Kullback-Leibler

Mathematica notebook addressing discrete entropy, conditional entropy, mutual information, and Kullback-leibler distance. Get/View PDF, Get Notebook.

The AEP and the Typical Set

Mathematica notebook with nice examples of the typical set. Get/View PDF, Get Notebook.

Huffman Codes

Java-based Huffman code demos from: Western Australia, and Texas Tech.
Mathematica Notebooks that explore the Huffman technique. Get Notebook 1, Get Notebook 2.
MatLab:
huffman.m - MATLAB huffman coder -- [J. Schamus]
huffman2.m - more MATLAB huffman coding -- [J. Schamus]
huffman1.m - MATLAB Huffman routines [R. Vogtli]
huffman.zip - More MATLAB Huffman routines.

Lempel-Ziv-Welch Codes

Java-based demo from Simon Fraser Univ.

Differential Entropy

Mathematica Notebook that defines and explores the differential entropy function. Get/View PDF, Get Notebook.
MatLab: Here's nothing for now.
Paul Viola's dissertation - A novel method for estimating continuous probability distributions from data.
Mathematica Notebooks that explore the Viola technique. Get/View PDF, Get/View PDF.

ASCII Reading Resources

Here's a page that contains assorted ASCII reading procedures.

Image Resources

image histograms - relative frequencies for colors in images -- [D. Hoeferlin]
imtools - Image Tools from San Diego Supercomputing Center

Homework Assignments and other Handouts:

The finer print:
Files available on this page are in several formats. Mathematica Notebook (.nb) files may be downloaded and used while running Mathematica or viewed with Mathematica Player (free) , both available from Wolfram Research. Matlab files (.m files) are in plain text and may be downloaded directly. In addition, some documents are available in Adobe PDF format (.pdf). PDF files may be viewed using Acrobat Reader. This utility is free from Adobe and runs on most anything: PC's, MAC's, many UNIX boxes.

Revised 2008.01 by Fred Garber, fred.garber@wright.edu