An example run of the multivariate program is in run_multivariate_mi.sh

The output of the multivariate program consists, in order, of

- the tuple specification as specified in the input specification using the -tupleFile switch,
- the corresponding MI or entropy value depending on whether two sets of tuples are specified or only a single one respectively,
- the # of data samples used, 
- the # of EM iterations
- the entropies of the X, Y and (X,Y) sets respectively in the MI case

If the -o switch is supplied output goes to the specified file,
otherwise it goes to standard output.

A discrete version of the program is in the subdirectory discrete-mi,
which also contains a simple example script.

Inputs are: (1) a GMTK-style observation file (with similar input
options that GMTK accepts), and (2) tuple-file which specifies the
offsets for which the MI needs to be calculated.

As an example, suppose you have three features and a label and you
store in a GMTK observation file such that the first three elements
are the features and the last one the label.  To compute the MI
between each feature and the label, you'd use the tuple file below:


[0@0] [3@0]
[1@0] [3@0] 
[2@0] [3@0]

assuming positions 0,1, and 2 for the features and index 3 for the
label.

This will produce 3 MI values I(0;3),I(1;3), and I(2;3), one for each
line of tuple file.

[0,1,2@0] [3@0] would compute I(0,1,2 ; 3), the joint MI between
(0,1,2) and the label 3.

The "@0" means that all features and the label are in the same time
frame relative to each other.  If you had a time series and wanted the
MI between, say feature 2 in the previous time frame and the current
one, you'd use 

[2@-1] [2@0]

The example ./discrete-mi/run_discrete_mi2.sh computes MI on some
discrete data in the subdirectory tuple-files using the 3-line tuple
file above.  It outputs 3 MI values in the file specified using the -o
option.

If your features are continuous and your label discrete you can
calculate H(F1,F2,F3) using a tuple line of the form

[F1,F2,F3@0] 

then calculate H(F1,F2,F3|LABEL) the same way as above except the
observations should only contain frames with label LABEL.

Finally I(F1,F2,F3;LABEL)=H(F1,F2,F3)-H(F1,F2,F3|LABEL)
