Information Theory and Coding

Course aim

The course presents in detail the theory and practice of efficient storage and transmission of information over noisy channels.

Course content

Review of Mathematical Background: Concepts from Probability Theory: Joint and Conditional Probability Distributions, Random Variables, Random Processes, and Markov Chains; Concepts from Mathematical Analysis: Continuity, Convexity, Concavity and Jensen’s inequality.
Review of Information Measures and their Properties: Entropy; Mutual Information; Conditional Entropy; Conditional Mutual Information; Kullback-Leibler Divergence; Log-sum inequality; Data Processing Inequality; Fano’s inequality.
Asymptotic Equipartition Property (AEP): Weak Law of Large Numbers; AEP as a consequence of the Weak Law of Large Numbers; Tail event bounding techniques; Types and Typicality; The lossless source coding theorem.
Lossy Compression: Motivation; Rate-Distortion (RD) Theory for Discrete Memoryless Sources (Coding and Converse).
Reliable Communication over Noisy Channels: Discrete memoryless channels; Codes, Rates, Redundancy and Reliable communication; Shannon’s Channel Coding Theorem and its Converse; Joint Source-Channel Coding.
Fundamentals of Linear Codes: Motivation; Introduction to Error Detection and Correction; Types of Codes; Minimum Distance; Vector Algebra; Block Coding Principles: Generator Matrix Description, Systematic Codes, Detection and Correction Bounds.

Convolutional Codes: encoders; generator polynomials; constraint length; state diagrams; tree and trellis diagrams; distance measures.

Decoding Convolutional Codes: Viterbi algorithm; Soft-Decision Decoding; MAP Decoding.

LDPC Codes: Code Density, Tanner Graphs; Decoding on Graphs: Message Passing and Belief Propagation.

Textbook(s)

Cover, Thomas M. 2006, Elements of Information Theory, 2nd, Wiley-Interscience

Lin, Shu; Costello, D. J. 2004, Error Control Coding, 2nd, Prentice-Hall, New Jersey

Prerequisite(s)

A good understanding of probability theory is required. Knowledge of communication theory would be advantageous.

Corequisite(s)

Nil

Teaching method

Note: These components may or may not be scheduled in every study period. Please refer to the timetable for further details.

Assessment

Assignment in 2 parts, Computer modelling project - code design, Examination

Fees

EFTSL*: 0.125
Commonwealth Supported program (Band 2)
To determine the fee for this course as part of a Commonwealth Supported program, go to:
How to determine your Commonwealth Supported course fee. (Opens new window)

Fee-paying program for domestic and international students
International students and students undertaking this course as part of a postgraduate fee paying program must refer to the relevant program home page to determine the cost for undertaking this course.

Non-award enrolment
Non-award tuition fees are set by the university. To determine the cost of this course, go to:
How to determine the relevant non award tuition fee. (Opens new window)

Not all courses are available on all of the above bases, and students must check to ensure that they are permitted to enrol in a particular course.

* Equivalent Full Time Study Load. Please note: all EFTSL values are published and calculated at ten decimal places. Values are displayed to three decimal places for ease of interpretation.