Information Theory and Coding

Course aim

To introduce the basic tools and concepts in information theory, and practical coding schemes for data compression and transmission over noisy channels.

Course content

The first part of this course is about information theory. The course aims at answering two fundamental questions in data communications:

- What is the ultimate data compression?

- What is the ultimate transmission rate of a communication channel?

To answer these questions, the basic tools and concepts in information theory are introduced. Practical coding schemes for data compression and transmission over noisy channels are introduced in the second part of this course. Review of Mathematical Background and Basic Concepts in Probability Theory.

Introduction of Information Measures and their Properties: Entropy; Mutual Information; Conditional Entropy; Conditional Mutual Information; Kullback-Leibler Divergence; Log-sum inequality; Data Processing Inequality; Fano’s inequality.

Asymptotic Equipartition Property (AEP): Law of Large Numbers; Types and Typicality.

Lossless source coding theorem and algorithms.

Noisy Channel Models and Shannon’s Channel Coding Theorem.

Fundamentals of Linear Codes: Introduction to Error Detection and Correction; Types of Codes; Minimum Distance.

Block Coding Principles: Generator Matrix Description, Systematic Codes, Detection and Correction Bounds.

Convolutional Codes (Encoding): encoders; generator polynomials; constraint length; state diagrams; tree and trellis diagrams; distance measures.

Convolutional Codes (Decoding): Viterbi algorithm; Soft-Decision Decoding; MAP Decoding.

LDPC Codes: Code Density, Tanner Graphs, Belief Propagation.

Special topics: Network Coding and Digital Fountain.

Textbook(s)

Cover, Thomas M. 2006, Elements of Information Theory, 2nd, Wiley-Interscience

Lin, Shu; Costello, D. J. 2004, Error Control Coding, 2nd, Prentice-Hall, New Jersey

Prerequisite(s)

A good understanding of probability theory is required. Knowledge of communication theory would be advantageous.

Corequisite(s)

Nil

Teaching method

Component	Duration
INTERNAL, MAWSON LAKES
Lecture	2 hours x 13 weeks
Tutorial	1 hours x 13 weeks

Note: These components may or may not be scheduled in every study period. Please refer to the timetable for further details.

Assessment

Examination, Problem solving exercise, Programming exercise

Fees

EFTSL*: 0.125
Commonwealth Supported program (Band 2)
To determine the fee for this course as part of a Commonwealth Supported program, go to:
How to determine your Commonwealth Supported course fee. (Opens new window)

Fee-paying program for domestic and international students
International students and students undertaking this course as part of a postgraduate fee paying program must refer to the relevant program home page to determine the cost for undertaking this course.

Non-award enrolment
Non-award tuition fees are set by the university. To determine the cost of this course, go to:
How to determine the relevant non award tuition fee. (Opens new window)

Not all courses are available on all of the above bases, and students must check to ensure that they are permitted to enrol in a particular course.

* Equivalent Full Time Study Load. Please note: all EFTSL values are published and calculated at ten decimal places. Values are displayed to three decimal places for ease of interpretation.