October 29, 2013

When doing ancestral sampling of the posterior, each curve's conditional posterior depends on (a) it's relevant (noisy) observation and (b) the sampled (noise-free) values of the parent. We can incorporate both in the posterior by treating the noise-free values as observations with zero variance in the likelihood.

In practice, theres a minor issue implementing this. Recall that because 3D observations are degenerate in one direction (namely the backprojection direction), we prefer to work with the precision matrix, \(\Lambda\), instead of a covariance matrix. Under this representation, noise-free values have infinite precision, so operations on \(\Lambda\) are invalid. Instead, we take a hybrid appraoch, using both precisions and covariances.

Recall the standard formulation for the posterior mean:

\[
\mu = K_* \left[ K + \Sigma \right]^{-1} y
\]

Without loss of generality, assume zero-noise observations appear after noisy observations. We can rewrite the posterior in terms of the precision matrix \(\Lambda\) of the noisy values:

\[
\begin{align}
\mu &= K_* \left[ K + \left(
\begin{array}{cc}
\Lambda^{-1} & 0 \\
0 & 0
\end{array}
\right) \right]^{-1} y \\
&= K_* \left[
\left(
\begin{array}{cc}
\Lambda & 0 \\
0 & I
\end{array}
\right)
K +
\left(
\begin{array}{cc}
I & 0 \\
0 & 0
\end{array}
\right)
\right]^{-1}
\left(
\begin{array}{cc}
\Lambda & 0 \\
0 & I
\end{array}
\right)
y \\
\end{align}
\]

We can implement this by slightly modifying our code for the precision-matrix formulation of posterior. First, give all noise-free values a precision of 1.0 in \(\Lambda\), and then modify the \(I\) inside the parentheses by zeroing-out elements corresponding to noise-free values. Using prime symbol to denote the modified matricies, the result is

\[
\mu = K_* \left[ \Lambda' K + I' \right]^{-1} \Lambda' y
\]

The expression for posterior covariance is derived in the same way, and has similar form.

Posted by Kyle Simek
Ancestral Sampling from poserior (Markov-aware sampling) →
← Implementing Nystrom method; bugs in posterior sampling code