Quantcast
Viewing all articles
Browse latest Browse all 10
↧

Visualizing the mutual information and an introduction to information geometry

July 1, 2012, 3:53 pm
$
0
0

For a while now I have had an interest in information geometry. The maxims that geometry is intuitive maths and information theory is intuitive statistics seem pretty fair to me, so it’s quite surprising to find a lack of easy to understand introductions to information geometry. This is my first attempt, the idea is to get an geometric understanding of the mutual information and to introduce a few select concepts from information geometry.

Mutual information

Most people that have heard of it will know that the mutual information is a measurement of how correlated two variables are. It is usually written as:

Image may be NSFW.
Clik here to view. MI(X,Y)=-\sum_x \sum_y p(x,y) \log \frac{p(x)p(y)}{p(x,y)}

but it can also be written as a kind-of-distance-measure (a Kullback-Leibler divergece) between a joint distribution Image may be NSFW.
Clik here to view. p(x,y) and a distribution where the probabilities of each Image may be NSFW.
Clik here to view. x or Image may be NSFW.
Clik here to view. y are the same as in Image may be NSFW.
Clik here to view. p(x,y) but the variables X and Y are independent; in other words, to the product of the marginal distributions Image may be NSFW.
Clik here to view. p(x)p(y). The distance measure, the Kullback-Leibler divergence between distributions Image may be NSFW.
Clik here to view. P and Image may be NSFW.
Clik here to view. Q is usually written Image may be NSFW.
Clik here to view. D_{KL}(P||Q) and is equal to (for discrete variables)

Image may be NSFW.
Clik here to view. D_{KL}(P||Q) = -\sum_x p(x) \log \frac{q(x)}{p(x)}

Which allows us to write the mutual information as

Image may be NSFW.
Clik here to view. MI(X,Y)=D_{KL}(p(x,y)||p(x)p(y))

with the sum in Image may be NSFW.
Clik here to view. D_{KL} being over both Image may be NSFW.
Clik here to view. x and Image may be NSFW.
Clik here to view. y.

For the example here I will look at the mutual information between two binary variables; Image may be NSFW.
Clik here to view. \mathcal{A}: which can be either Image may be NSFW.
Clik here to view. A or Image may be NSFW.
Clik here to view. \neg A, and Image may be NSFW.
Clik here to view. \mathcal{B}: which can be either Image may be NSFW.
Clik here to view. B or Image may be NSFW.
Clik here to view. \neg B. In this case there is only four different probabilities to consider; as these must sum to one, there are only three degrees of freedom and we can visualize the probability space in three dimensions.

The standard geometry bit

To begin with it is useful to introduce two coordinate systems. Firstly, one that is a ‘natural coordinate system’ in a sense that I will explain later (the Image may be NSFW.
Clik here to view. i in Image may be NSFW.
Clik here to view. \xi^i is a superscript index not a power):

Image may be NSFW.
Clik here to view. p(A,B) = \xi^1
Image may be NSFW.
Clik here to view. p(A,\neg B) = \xi^2
Image may be NSFW.
Clik here to view. p(\neg A,B) = \xi^3
Image may be NSFW.
Clik here to view. p(\neg A,\neg B) = 1-\xi^1-\xi^2-\xi^3

in this coordinate system Image may be NSFW.
Clik here to view. p(A) = p(A,B)+p(A,\neg B) = \xi^1+\xi^2 and Image may be NSFW.
Clik here to view. p(B) = p(A,B)+p(\neg A,B) = \xi^1+\xi^3. The other coordinate system that I will use is one for visualizing things in Cartesian coordinates:

Image may be NSFW.
Clik here to view. x = 1-\xi^1-\xi^2 = 1-p(A) = p(\neg A)
Image may be NSFW.
Clik here to view. y = 1-\xi^1-\xi^3 = 1-p(B) = p(\neg B)
Image may be NSFW.
Clik here to view. z = 1-\xi^2-\xi^3 = 1-p(A \neq B) = p(A = B)

Image may be NSFW.
Clik here to view. x, Image may be NSFW.
Clik here to view. y and Image may be NSFW.
Clik here to view. z correspond to probabilities (as do the Image may be NSFW.
Clik here to view. \xis) so every probability must be inside the cube with edges Image may be NSFW.
Clik here to view. [0,1], but the space of probabilities is more constrained than this and in fact lies within a tetrahedron made out of alternate corners of this cube.

Image may be NSFW.
Clik here to view.

The cases where the variables Image may be NSFW.
Clik here to view. \mathcal{A} and Image may be NSFW.
Clik here to view. \mathcal{B} are independent form a surface in the tetraderon. We can get an equation for it from Image may be NSFW.
Clik here to view. p(A,B) = p(A)p(B), in the Cartesian coordinates we have:

Image may be NSFW.
Clik here to view. p(A)p(B) = p(A,B)
Image may be NSFW.
Clik here to view. (1-x)(1-y) = z - xy
Image may be NSFW.
Clik here to view. 1-x-y+2xy = z

For the natural coordinates we have:
Image may be NSFW.
Clik here to view. p(A)p(B) = p(A,B)
Image may be NSFW.
Clik here to view. (\xi^1+\xi^2)(\xi^1+\xi^3) = \xi^1

The surface looks like this:

Image may be NSFW.
Clik here to view. Image may be NSFW.
Clik here to view. Image may be NSFW.
Clik here to view.

There is also a set of lines formed by all the distributions that have the same marginal distributions (Image may be NSFW.
Clik here to view. p(A) and Image may be NSFW.
Clik here to view. p(B)). These are lines that are parallel with the Image may be NSFW.
Clik here to view. z-axis. Putting all this together we get a picture that looks like this:

Image may be NSFW.
Clik here to view.

Each line intersects with the surface exactly once. Which we can see when we look at it along the Image may be NSFW.
Clik here to view. z-axis. You can see that there is an intersection for every possible value of Image may be NSFW.
Clik here to view. x (Image may be NSFW.
Clik here to view. 1-p(A)) and Image may be NSFW.
Clik here to view. y (Image may be NSFW.
Clik here to view. 1-p(B)).

Image may be NSFW.
Clik here to view.

Arc lengths (Euclidean distances between points)

The mutual information, as we will see, can be thought of as measuring a distance to the curved surface along one of the lines in Image may be NSFW.
Clik here to view. z. This can be done in standard geometry, although it does not yield a standard information measure. The basic idea is, that to measure the length of a curve we add up lots of little bits of a curve.

Image may be NSFW.
Clik here to view.

The picture shows the case of a 2D Euclidean geometry with a curve between Image may be NSFW.
Clik here to view. A and Image may be NSFW.
Clik here to view. B in a plane with coordinates Image may be NSFW.
Clik here to view. x and Image may be NSFW.
Clik here to view. y. We can work out the length of the curve by integrating Image may be NSFW.
Clik here to view. ds along the curve – Image may be NSFW.
Clik here to view. ||s|| = \int_A^B ds. Another way of writing this is roughly:

Image may be NSFW.
Clik here to view. ||s|| = \int_A^B \sqrt{dx^2 + dy^2}

The standard thing to do is to write the coordinates of the curve as a function of some arbitrary measure (Image may be NSFW.
Clik here to view. t) of how far along the curve we have gone, usually we make it so that the parameters is Image may be NSFW.
Clik here to view. 0 at the start and Image may be NSFW.
Clik here to view. 1 at the end:

Image may be NSFW.
Clik here to view. (x,y) = (x(t),y(t))
Image may be NSFW.
Clik here to view. A = (x(0),y(0))
Image may be NSFW.
Clik here to view. B = (x(1),y(1))

Then we can write the integral as

Image may be NSFW.
Clik here to view. \int_A^B ds = \int_0^1 \sqrt{\left(\frac{\partial x(t)}{\partial t} dt\right)^2 + \left(\frac{\partial y(t)}{\partial t} dt\right)^2 } = \int_0^1 \sqrt{\left(\frac{\partial x(t)}{\partial t}\right)^2 + \left(\frac{\partial y(t)}{\partial t} \right)^2 } dt

To make this look a bit more like differential geometry we can write the bit in the square root in a different way. If we now number Image may be NSFW.
Clik here to view. x and Image may be NSFW.
Clik here to view. y as dimensions Image may be NSFW.
Clik here to view. 1 and Image may be NSFW.
Clik here to view. 2 and give them new names Image may be NSFW.
Clik here to view. x = q^1 and Image may be NSFW.
Clik here to view. y = q^2 (the raised numbers are just there to say which one is which, i.e. Image may be NSFW.
Clik here to view. q^2 does not mean ‘q squared’, but ‘q number 2′). Expressing the bit in the square root in these terms gives:

Image may be NSFW.
Clik here to view. \left(\frac{\partial x}{\partial t}(t) dt\right)^2 + \left(\frac{\partial y}{\partial t}(t) dt \right)^2 = \left(\frac{\partial q^1}{\partial t}(t)dt\right)^2 + \left(\frac{\partial q^2}{\partial t}(t)dt \right)^2
Image may be NSFW.
Clik here to view. = \sum_i \sum_j \delta_{ij}\frac{\partial q^i}{\partial t}(t)\frac{\partial q^j}{\partial t}(t)dt^2 = \sum_i \sum_j \delta_{ij} dq^i dq^j

I have introduced a new thing here, the $\delta$. This is basically the identity matrix, it takes the value Image may be NSFW.
Clik here to view. 1 when Image may be NSFW.
Clik here to view. i=j and Image may be NSFW.
Clik here to view. 0 when Image may be NSFW.
Clik here to view. i \neq j. The benefit of doing this that we change the coordinates to something different and keep the distance the same by changing $\delta_{ij}$ for something else. For example if we have new coordinates Image may be NSFW.
Clik here to view. (m^1, m^2) that are related to Image may be NSFW.
Clik here to view. (q^1, q^2) by Image may be NSFW.
Clik here to view. m^1 = (q^1)^2 (read ‘q one squared’) and Image may be NSFW.
Clik here to view. m^2 = (q^2)^3 (read ‘q two cubed’) then we replace Image may be NSFW.
Clik here to view. q with Image may be NSFW.
Clik here to view. m and Image may be NSFW.
Clik here to view. \delta_{ij}(q) with a new quantity Image may be NSFW.
Clik here to view. g_{ij}(m) which is given by (I won’t show the working):

Image may be NSFW.
Clik here to view. g_{1,1} = \frac{1}{4} (m^1)^{-1}
Image may be NSFW.
Clik here to view. g_{2,2} = \frac{1}{9} (m^2)^{-\frac{4}{3}}
Image may be NSFW.
Clik here to view. g_{1,2} = g_{2,1} = 0

The quantity Image may be NSFW.
Clik here to view. g is known as the metric tensor (or the coordinate based representation of it). Ultimately, it is the quantity that defines the distance between points in Riemannian geometry. When we change coordinates we have to change the value of Image may be NSFW.
Clik here to view. g so as to keep distances stay the same. Really, it is the relationship between Image may be NSFW.
Clik here to view. g and the coordinates that defines distances. I’m going to skip how do the coordinate transformations but you can find a description here – it’s not particularly hard but I’ve waffled about this a bit too much already.

So, in general, the distance (arc-length) between two points can be written as:

Image may be NSFW.
Clik here to view. \int_A^B \sqrt{\sum_i \sum_j g_{ij}dq^idq^j} or Image may be NSFW.
Clik here to view. \int_A^B \sqrt{\sum_i \sum_j g_{ij}\frac{\partial q^i}{\partial t}\frac{\partial q^j}{\partial t}}dt

Later, it will be useful if we call the bit in the square root Image may be NSFW.
Clik here to view. G(t) so that:

Image may be NSFW.
Clik here to view. \int_A^B \sqrt{G(t)} dt

The metric tensor and divergences

When the Kullback-Liebler divergence was first published, they noticed that approximately (for small Image may be NSFW.
Clik here to view. \Delta\xi) that for distributions with parameters Image may be NSFW.
Clik here to view. \xi, written here as Image may be NSFW.
Clik here to view. p(x;\xi):

Image may be NSFW.
Clik here to view. D_{KL}(p(x; \xi) || p(x; \xi + \Delta\xi)) \approx \frac{1}{2}\sum_i \sum_j f_{ij}(\xi) \Delta\xi^i \Delta\xi^j

where Image may be NSFW.
Clik here to view. f_{ij}(\xi) is the Fisher information, it is a metric tensor (a valid Image may be NSFW.
Clik here to view. g_{ij}) which can be calculated by:

Image may be NSFW.
Clik here to view. f_{ij}(\xi) = \sum_x p(x;\xi) \frac{\partial}{\partial \xi^i} \log{p(x;\xi)} \frac{\partial}{\partial \xi^j} \log{p(x;\xi)}
or equally (though not obviously)
Image may be NSFW.
Clik here to view. f_{ij}(\xi) = - \sum_x p(x;\xi) \frac{\partial}{\partial \xi^i} \frac{\partial}{\partial \xi^j} \log{p(x;\xi)}

This can be seen from the Taylor expansion, which is a bit of a pain to do, but quite straight forwards. Indeed, we can look at higher order terms too:

Image may be NSFW.
Clik here to view. D_{KL}(p(x; \xi) || p(x; \xi + d\xi)) = \frac{1}{2}\sum_i \sum_j f_{ij}(\xi) d\xi^i d\xi^j + \frac{1}{6}\sum_i \sum_j \sum_k h_{ijk}(\xi) d\xi^i d\xi^j d\xi^k + o(|d\xi|^4)

It turns out, for the Kullback-Liebler divergence (among others) that if we choose the coordinates Image may be NSFW.
Clik here to view. \xi correctly, then we can recover the divergence by integrating. Basically, we can do this if Image may be NSFW.
Clik here to view. h_{ijk} = \frac{\partial}{\partial \xi^k} g_{ij}. There are a number of ways of checking this, which can be found in this book, the simplest is checking that:
Image may be NSFW.
Clik here to view. \frac{\partial}{\partial \xi^i}\frac{\partial}{\partial \xi^j} p(x;\xi) = 0
or for the reverse Kullback-Liebler divergence:
Image may be NSFW.
Clik here to view. \frac{\partial}{\partial \xi^i}\frac{\partial}{\partial \xi^j} \log{p(x;\xi)} = 0
for all Image may be NSFW.
Clik here to view. x.

It’s like a half square arc-length

From the section on classical geometry, we can see that small arc-lengths Image may be NSFW.
Clik here to view. \Delta s are approximately given by:

Image may be NSFW.
Clik here to view. \Delta s \approx \sqrt{G(t)}\Delta t

But the divergences are approximately given by

Image may be NSFW.
Clik here to view. \Delta D \approx \frac{1}{2}G(t)\Delta t^2

So, the divergence is a kind of half-square-arc-length. But the actual squared arc-length is something different; we can see this by direct calculation:

Image may be NSFW.
Clik here to view. \frac{1}{2}s^2 = \frac{1}{2} \left( \int_A^B \sqrt{G(s)}ds \right) \left( \int_A^B \sqrt{G(t)} dt \right) = \frac{1}{2} \int_A^B \int_A^B \sqrt{G(s)G(t)} ds dt

We can visualize this as an integral over a square with sides of Image may be NSFW.
Clik here to view. s and Image may be NSFW.
Clik here to view. t. If we represent the magnitude of Image may be NSFW.
Clik here to view. \sqrt{G} as the intensity of a greenness within the square we represent the area that we are integrating to get the square arc-length as:

Image may be NSFW.
Clik here to view.

Notice the symmetry in the pattern that is formed, it is symmetric in the axis where Image may be NSFW.
Clik here to view. s = t. This makes it possible to write the half-square arc-length as the integral over a triangle:

Image may be NSFW.
Clik here to view. \int_{A \leq s \leq t \leq B} \sqrt{G(s)G(t)} ds dt

Which is symmetric in the sense that it stays the same if we swap Image may be NSFW.
Clik here to view. s and Image may be NSFW.
Clik here to view. t. We can view this integral in the same way as before.

Image may be NSFW.
Clik here to view.

The divergence on the other hand is calculated by (where Image may be NSFW.
Clik here to view. b is the value of Image may be NSFW.
Clik here to view. s at Image may be NSFW.
Clik here to view. B):
Image may be NSFW.
Clik here to view. D = \int_{A \leq s \leq t \leq B} g(s) ds dt = \int_A^B (b-s) g(s) ds
which is asymmetric, there is a ‘forwards’ and ‘backwards’ integral. It looks like

Image may be NSFW.
Clik here to view.

The asymmetry of the divergence is well known. The breaking of this symmetry corresponds to the fundamental difference between information measures and normal geometric measures.

We are now at a point where we can use an information theoretic integral instead of a geometric one to calculate the distance between points. This method has (nearly) all the features of differential geometry, but the quantities are those of information theory.

Back to the Mutual Information

So, let us go back to the description of the mutual information. We know that it is Kullback-Leibler from the joint distribution to the a point with the same Image may be NSFW.
Clik here to view. x and Image may be NSFW.
Clik here to view. y coordinates. But also, we can say a little more. Lines parallel with the Image may be NSFW.
Clik here to view. z-axis are straight lines in terms of divergences (unlike normal geometry, this is not when the metric is constant). We can see this by checking that

Image may be NSFW.
Clik here to view. \frac{\partial}{\partial z} \frac{\partial}{\partial z} p(a,b; z) = 0

for all Image may be NSFW.
Clik here to view. a and Image may be NSFW.
Clik here to view. b. This is just the check I mentioned before. This holds because the Image may be NSFW.
Clik here to view. x, Image may be NSFW.
Clik here to view. y and Image may be NSFW.
Clik here to view. z are linearly related to the probabilities. The result of this is that we can integrate Fisher information in a straight line along the Image may be NSFW.
Clik here to view. z-axis in the Euclidean coordinates and get the Kullback-Leibler divergence.

In other words we can write that the mutual information, for a probability written in terms of Image may be NSFW.
Clik here to view. (x,y,z):

Image may be NSFW.
Clik here to view. \int_{1-x-y+2xy}^z \int_t^z f(s) dt ds = \int_{1-x-y+2xy}^z (z-s) f(s) ds

where the Fisher information metric, Image may be NSFW.
Clik here to view. f(z), is given by:

Image may be NSFW.
Clik here to view. f(z) = \frac{8}{1-x+y+z} - \frac{2}{x-y+z+1} - \frac{2}{-x+y+z-1}

The fact that the straight lines for the divergence are straight lines in the Cartesian space is important. In a very real sense, we are only comparing the joint distribution (Image may be NSFW.
Clik here to view. p(a,b)) with others with different Image may be NSFW.
Clik here to view. z value (different Image may be NSFW.
Clik here to view. p(A=B) – different degrees of independence – but with the same value of Image may be NSFW.
Clik here to view. p(A) and Image may be NSFW.
Clik here to view. p(B)). Roughly, we could do anything to the geometry of the space that isn’t on the Image may be NSFW.
Clik here to view. z line and the divergence would be unaffected.

I think that’s about all I have to say for now. I’m working on a more detailed tutorial for information geometry in general, I’ll post a link when I have finished it.


Image may be NSFW.
Clik here to view. Image may be NSFW.
Clik here to view.
↧
Search
Viewing all articles
Browse latest Browse all 10

Trending Articles

VHSE First (1st) Allotment 2025 - vhscap.kerala.gov.in

May 24, 2025, 11:58 pm

Bureau of Internal Revenue: Regional Offices (Directory)

January 9, 2014, 11:06 pm

Game Of Thrones S03 Season 3 720p BluRay DTS x264-PublicHD

February 16, 2014, 4:15 am

Black Angus Grilled Artichokes

July 16, 2016, 4:37 pm

Moondru Mudichu 20-07-2016 – Polimer tv Serial

July 20, 2016, 9:25 am

Best Suvichar in Hindi |बेस्ट सुविचार |शुभ विचार हिंदी में

March 7, 2020, 11:19 pm

Connect failed:(10060) The current connection has timeout.

July 27, 2009, 2:34 am

Lady Gaga – MAYHEM (Bonus Tracks Version) [iTunes Rip M4A]

March 7, 2025, 11:07 am

Notes of Development of Phy. Edu. - Post Independence| Class 11th Physical...

May 4, 2016, 1:02 am

Tagalog Praise and Worship Song Lyrics with Chords

October 21, 2009, 4:44 am

Tumkur University Results 2017 May-June Semester 1st-2nd-3rd-4th-5th-6th

July 3, 2017, 5:07 am

VARRIO KING KOBRAS RIFA

June 25, 2017, 11:10 pm

Practice Sheet of Right form of verbs for HSC Students

September 22, 2019, 11:40 pm

Windows Update / Microsoft Update の接続先 URL について

February 27, 2017, 12:32 am

Forum Post: RE: TSW14J58EVM: TSW12QJ1600EVM interfacing with TSW14J58EVM...

May 22, 2024, 1:37 pm

A/L Technology Stream – Subject combinations, Syllabuses and Teacher guides

December 17, 2013, 6:12 pm

How The Feds Finally Toppled Detroit Mafia Czar Tony Giacalone

June 19, 2015, 11:18 pm

The Angry Birds Movie (Tamil Dubbed)

May 29, 2016, 1:08 am

Tench receives death penalty

April 25, 2016, 11:38 am

Late educator remembered at graduation

July 2, 2016, 9:44 pm
Search