Mutual Information

Mutual Information and Dependence

Assumed audience

• Reading level: technical.
• Background: probability and basic entropy.
• Goal: understand mutual information as shared uncertainty.

Definition

Mutual information measures how much knowing one variable reduces uncertainty about another:

[ I(X;Y) = H(X) + H(Y) - H(X,Y). ]

Interpretation

• If $X$ and $Y$ are independent, $I(X;Y)=0$.
• If $X$ determines $Y$, mutual information is high.

Why this matters

Mutual information is a core tool for dependency detection and feature selection.