hypertools.tools.format_data

hypertools.tools.format_data(x, vectorizer='CountVectorizer', semantic='LatentDirichletAllocation', corpus='wiki', ppca=True, text_align='hyper', impute=None)[source]

Formats data into a list of numpy arrays

This function is the standard input pass shared by hypertools’ analysis and plotting functions: it wraps the input into a list of 2-D (observations x features) float arrays, converting pandas DataFrames (binarizing text columns via df2mat) and embedding text (strings / lists of strings) into numeric matrices via text2mat along the way. Missing (NaN) values are filled via PPCA (or the impute= override), and when text and numeric datasets with matching sample counts are mixed, they are aligned into a common space.

Input conversions (release-1.0 audit):

  • pandas Series (top-level or inside a list) become 1-D datasets; tuples are treated like lists.

  • Nested lists/tuples of arrays/DataFrames (e.g. [[arr1, arr2]]) are flattened into a flat list of datasets, matching hyp.plot().

  • Lists of bools are numeric 0/1 datasets, like np.array([True, ...]).

  • numpy MaskedArray masked entries are treated as MISSING data: converted to NaN (with a UserWarning) so they flow into the standard PPCA/impute= fill, never analyzed as real values.

  • DataFrame Categorical columns are dummy-coded exactly like object-dtype string columns (via df2mat); datetime columns are converted to float seconds since the Unix epoch, with NaT becoming NaN and a UserWarning naming the converted columns.

Parameters:
xnumpy array, dataframe, series, string or (mixed, possibly nested) list

The data to convert

vectorizerstr, dict, class or class instance

The vectorizer to use. Built-in options are ‘CountVectorizer’ or ‘TfidfVectorizer’. String names resolve in scikit-learn -> gensim -> Hugging Face order (GH #198): unrecognized names are treated as Hugging Face sentence-transformers model ids, and if loading one fails, a ValueError names the offending value and the built-in options (so typos don’t surface as raw network errors). To change default parameters, set to a dictionary e.g. {‘model’ : ‘CountVectorizer’, ‘kwargs’ : {‘max_features’ : 10}} (the legacy {‘model’, ‘params’} form is also still accepted). See https://scikit-learn.org/stable/api/sklearn.feature_extraction.html for details. You can also specify your own vectorizer model as a class, or class instance. With either option, the class must have a fit_transform method (see https://scikit-learn.org/stable/data_transforms.html). To set parameters, use the dict form (or a configured class instance); a bare class is instantiated with its defaults.

semanticstr, dict, class or class instance

Text model to use to transform text data. Built-in options are ‘LatentDirichletAllocation’ or ‘NMF’ (default: LDA). To change default parameters, set to a dictionary e.g. {‘model’ : ‘NMF’, ‘kwargs’ : {‘n_components’ : 10}} (the legacy {‘model’, ‘params’} form is also still accepted). See https://scikit-learn.org/stable/api/sklearn.decomposition.html for details on the two model options. You can also specify your own text model as a class, or class instance. With either option, the class must have a fit_transform method (see https://scikit-learn.org/stable/data_transforms.html). To set parameters, use the dict form (or a configured class instance); a bare class is instantiated with its defaults.

corpuslist (or list of lists) of text samples or ‘wiki’, ‘nips’, ‘sotus’.

Text to use to fit the semantic model (optional). If set to ‘wiki’, ‘nips’ or ‘sotus’ and the default semantic and vectorizer models are used, a pretrained model will be loaded which can save a lot of time.

ppcabool

Performs PPCA to fill in missing values (default: True)

imputestr, dict, class, class instance or None

If missing (NaN) values are present and ppca is True, this overrides the default PPCA fill with a different hypertools.impute model (e.g. ‘Kalman’, ‘KNNImputer’; see hypertools.impute.impute for accepted forms). If None (default), missing values are filled with PPCA, matching the pre-1.0 behavior byte-for-byte.

text_alignstr

Alignment algorithm to use when both text and numerical data are passed. If numerical arrays have the same shape, and the text data contains the same number of samples, the text and numerical data are automatically aligned to a common space. Example use case: an array of movie frames (frames by pixels) and text descriptions of the frame. In this case, the movie and text will be automatically aligned to the same space (default: hyperalignment).

Returns:
datalist of numpy arrays

A list of formatted arrays