hypertools.describe

hypertools.describe(x, reduce='IncrementalPCA', max_dims=None, show=True, format_data=True, backend='auto')

Describe how well reduced data preserves the raw data’s pairwise-distance structure, as a function of the number of dimensions

For each candidate number of dimensions, this function reduces the data and Pearson-correlates the flattened pairwise Euclidean distance matrix of the reduced data with that of the raw (full-dimensional) data, to give a sense for how well the data can be summarized with n dimensions. Useful for evaluating quality of dimensionality reduced plots.

Parameters:

xNumpy array, DataFrame or list/tuple of arrays/dfs

A list of Numpy arrays or Pandas Dataframes (a tuple of datasets is treated exactly like a list). Datasets in a list are stacked into one shared feature space, so they must all have the same number of columns (ragged lists raise a ValueError); None raises a TypeError.

reducestr, dict, class, instance, or fitted Reducer

Decomposition/manifold learning model to use (default: ‘IncrementalPCA’). Models supported: PCA, IncrementalPCA, SparsePCA, MiniBatchSparsePCA, KernelPCA, FastICA, FactorAnalysis, TruncatedSVD, DictionaryLearning, MiniBatchDictionaryLearning, TSNE, Isomap, SpectralEmbedding, LocallyLinearEmbedding, MDS, and UMAP; the mixture models GaussianMixture, BayesianGaussianMixture, LatentDirichletAllocation and NMF (GH #174); and the torch autoencoders Autoencoder, DeepAutoencoder, SparseAutoencoder, ConvolutionalAutoencoder, SequenceAutoencoder and VariationalAutoencoder (GH #162, pip install “hypertools[torch]”). Can be passed as a string, or for finer control as a dictionary, e.g. reduce={‘model’: ‘PCA’, ‘kwargs’: {‘whiten’: True}}. See scikit-learn model docs for details on parameters supported for each model.

max_dimsint

Dimensionalities 2 through max_dims - 1 are evaluated (the bound is EXCLUSIVE, matching Python’s range), so max_dims must be an integer >= 3 (or None); anything else raises a ValueError naming the kwarg. Defaults to min(n_observations, n_features) of the stacked data (floored at 3, so 2-feature data still evaluates its one meaningful dimensionality; 1-feature or single-observation data raises a ValueError – there is no dimensionality structure to describe). Values beyond the data’s own dimensionality are clamped with a UserWarning – past min(n_observations, n_features) the correlations just flatline at 1.0, which is not evidence for more meaningful components. Note: with reduce=’TSNE’ (default barnes_hut method), only dimensionalities 2-3 can be evaluated – larger max_dims values are clamped with a UserWarning; pass reduce={‘model’: ‘TSNE’, ‘kwargs’: {‘method’: ‘exact’}} to evaluate more.

showbool

Plot the result (default: True). The figure is displayed only when the resolved backend can show one (plotly, or an interactive matplotlib backend); under a non-interactive matplotlib backend (e.g. Agg) the figure is still drawn and returned in the result dict’s ‘fig’ key, without calling plt.show().

format_databool

Whether or not to first call the format_data function (default: True).

backend{‘auto’, ‘matplotlib’, ‘plotly’}

Which plotting backend to draw the correlation-vs-dimensions figure with when show=True. Validated eagerly (even with show=False, an unknown backend raises the same “backend must be one of …” ValueError as hyp.plot). Default: ‘auto’, resolved the same way hyp.plot resolves it – plotly on Colab/Kaggle when available, else matplotlib. The matplotlib figure is a seaborn line plot with the top and right spines removed; the plotly figure is an interactive go.Figure (which has no top/right spines by default). Multi-dataset inputs get one distinguishable color per dataset, a legend, and the ‘average’ curve overlaid, on both backends. The analysis results in the returned dict are identical either way (only ‘fig’ differs: it holds the backend’s own figure object).

Returns:

resultdict

A dictionary with the analysis results. ‘average’ is the correlation by number of components for all data. ‘individual’ is a list of lists, where each list is a correlation by number of components vector (for each input list). ‘fig’ is the rendered figure handle (a matplotlib.figure.Figure or plotly.graph_objects.Figure, depending on the backend) when show=True and a figure was drawn; otherwise None.

Examples

>>> import numpy as np
>>> import hypertools as hyp
>>> x = np.cumsum(np.random.default_rng(0).standard_normal((40, 5)),
...               axis=0)
>>> result = hyp.describe(x, reduce='PCA', max_dims=4, show=False)
>>> sorted(result.keys())
['average', 'fig', 'individual']