[Work Log] FIRE - Refactoring, k-folds cross validation, baseline evaluation

May 23, 2014
Project FIRE
Subproject Piecewise Linear Clustering
Working path projects/​fire/​trunk/​src/​piecewise_linear
SVN Revision unknown (see text)
Unless otherwise noted, all filesystem paths are relative to the "Working path" named above.

Refactored code base to better support model evaluation and k-folds cross validation.

Run: Cross validation (baseline)

Description: Run 10-folds cross validation on the non-sampling (analytical) model.


Error values are mean per-dimension negative log-likelihood.

3 kmeans iterations:
training error: 3.94522
test error: 4.9166

5 kmeans iterations:
training error: 3.97775
test error: 4.9501

10 kmeans iterations:
training error: 4.01519
test error: 5.0074

20 kmeans iterations:
training error: 4.01538
test error: 5.0068

Increasing iterations also increases training error, which seems counterintuitive, especially because the algorithm debugging messages show the total log likelihood increasing.

But the log likelihood uses known membership, so values will naturally be higher. Model evaluation marginalizes over clusters, and since we have fixed the cluster weights to be equal, the dominating cluster is under-promoted. I would guess setting cluster weights from training membership proportions will reverse this trend, so more iterations will improve training error.

Run: Re-run with uneven cluster wieghts

Description: Set cluster weights proportional to the training membership values.


5 itns
training error: 3.62931
test error: 4.61624

10 itns
training error: 3.61348
test error: 4.60147

15 itns
training error: 3.60094
test error: 4.58815

20 itns
training error: 3.60094
test error: 4.58815

Discussion: Success! Error dropped as kmeans converged further

Posted by Kyle Simek
blog comments powered by Disqus