Interpolating missing data with probabalistic PCA

  run this example as a notebook — or grab the .ipynb (also linked at the bottom of the page)

When you pass a matrix with with missing data, hypertools will attempt to fill in the values using probabalistic principal components analysis (PPCA). Here is an example where we generate some synthetic data, remove some of the values, and then use PPCA to interpolate those missing values. Then, we plot both the original and data with missing values together to see how it performed.

/home/docs/checkouts/readthedocs.org/user_builds/hypertools/checkouts/latest/hypertools/plot/backend.py:1353: UserWarning: Failed to switch to any interactive backend (TkAgg, QtAgg, Qt5Agg, Qt4Agg, GTK4Agg, GTK3Agg, WXAgg). Falling back to 'Agg'.
  warnings.warn(BACKEND_WARNING)
/home/docs/checkouts/readthedocs.org/user_builds/hypertools/checkouts/latest/hypertools/tools/format_data.py:492: UserWarning: Missing data: filling missing values with PPCA (observed values are preserved exactly; only the NaN entries are reconstructed). Pass impute= to choose a different imputation model -- see hypertools.impute.
  warnings.warn('Missing data: filling missing values '

# Code source: Andrew Heusser
# License: MIT

# import
from scipy.linalg import toeplitz
import numpy as np
from copy import copy
import hypertools as hyp

# simulate data (seeded so the figure is reproducible)
np.random.seed(123)
K = 10 - toeplitz(np.arange(10))
data1 = np.cumsum(np.random.multivariate_normal(np.zeros(10), K, 250), axis=0)
data2 = copy(data1)

# simulate missing data: remove exactly 10% of the entries
missing = .1
inds = [(i,j) for i in range(data2.shape[0]) for j in range(data2.shape[1])]
missing_data = [inds[i] for i in np.random.choice(len(inds), int(len(inds)*missing), replace=False)]
for i,j in missing_data:
    data2[i,j]=np.nan

# plot
hyp.plot([data1, data2], linestyle=['-',':'], legend=['Original', 'PPCA'])