hypertools.load

hypertools.load(dataset, reduce=None, ndims=None, align=None, normalize=None, *, legacy=False, split=None, streaming=False, trust=False)[source]

Load data from a built-in example dataset, a scikit-learn or seaborn named dataset, a local file, a Hugging Face dataset, Google Drive, Dropbox, or any URL

A string is interpreted by trying, in order:

  1. a built-in example dataset name (listed below)

  2. a scikit-learn bundled dataset name – 'iris', 'digits', 'wine', 'breast_cancer', 'diabetes', or 'linnerud' (the small datasets shipped with sklearn.datasets; the network-fetched fetch_* datasets are not included). Loaded via the corresponding sklearn.datasets.load_* function and returned as a DataFrame of the features with the target appended as a 'target' column (for multi-output targets, e.g. 'linnerud', one column per target name instead)

  3. a seaborn dataset name – any name returned by seaborn.get_dataset_names() (e.g. 'penguins', 'tips', 'titanic'), loaded via seaborn.load_dataset() and returned unchanged. This is a network lookup (cached per-process); if it can’t reach the seaborn-data repo, this step is skipped

  4. a FiveThirtyEight dataset, explicit prefix 'fivethirtyeight/<slug>' (e.g. 'fivethirtyeight/bechdel'), where <slug> is the dataset’s folder in https://github.com/fivethirtyeight/data. The folder’s CSV file(s) are downloaded from raw.githubusercontent.com: a single CSV becomes a DataFrame, multiple CSVs become a dict of {filename: DataFrame}

  5. a Kaggle dataset, explicit prefix 'kaggle/<owner>/<dataset>' (e.g. 'kaggle/uciml/iris'), downloaded anonymously via kagglehub.dataset_download (requires the optional kagglehub dependency – pip install hypertools[kaggle]). Every CSV/TSV file in the dataset is loaded the same way as step 4

  6. a path to a local file (.geo/pickle, .npy/.npz, .csv/.tsv/.txt, .json, .parquet, .mat, .xlsx/.xls; gzip-compressed variants (.gz) are decompressed transparently). Files with no extension are parsed by content sniffing; files with any other extension raise an error unless their content matches a recognized binary format (pickle/npy/zip)

  7. a Hugging Face dataset id such as 'scikit-learn/iris' (pass streaming=True for a streaming dataset, which can be passed straight to hypertools.plot())

  8. a Google Sheets URL (docs.google.com/spreadsheets/d/<id>), loaded via its CSV export

  9. a Google Drive URL or bare file id (large files behind Drive’s “can’t scan this file for viruses” interstitial are followed automatically)

  10. a Dropbox URL or shared-link path

  11. any other URL, with or without an https:// scheme

Note

Precedence: a built-in example dataset name (step 1) always wins, even over a same-named scikit-learn/seaborn dataset. Between scikit-learn and seaborn, scikit-learn wins – e.g. 'iris' resolves to scikit-learn’s load_iris (columns like 'sepal length (cm)'), not seaborn’s 'iris' dataset (columns like 'sepal_length'), since both define an 'iris' name. Because these resolvers run before local-file resolution, a local file whose name (without an extension) matches a scikit-learn or seaborn dataset name is shadowed – pass a path with an extension, or an absolute/relative path containing a /, to force local-file resolution.

The 'fivethirtyeight/' and 'kaggle/' prefixes (steps 4-5) are explicit: a name starting with one of them is always treated as that source, so a same-named relative local path (e.g. a local file fivethirtyeight/bechdel) is shadowed – prepend './' to force local-file resolution. For the same reason, a prefixed name that then fails (unknown slug/dataset id, no CSV/TSV files found, malformed id) raises immediately instead of falling through to the remaining steps.

Parameters:
datasetstring, path-like, or list of strings

The name of a built-in example dataset (listed below), a dataset name resolvable per the steps above, or a file path / URL.

weights is a list of numpy arrays, one PER SUBJECT (36 arrays, each 300 timepoints x 100 parameters, float32), containing brain activity (fMRI) from subjects listening to the same story, fit using Hierarchical Topographic Factor Analysis (HTFA) with 100 nodes; each array’s rows are timepoints and its columns are model parameters.

weights_sample is a sample of 3 subjects from that dataset.

weights_avg is a 2-array group-averaged variant of the same experiment: a list of two (100, 100) arrays, one per group.

spiral is a list of two (1000, 3) numpy arrays containing 3D spiral data, used to highlight the procrustes function.

mushrooms is a pandas DataFrame of categorical features (columns) describing 8,124 mushroom samples (rows).

sotus is a list of 29 State of the Union addresses (1989-2018), as strings.

wiki is a list holding one (3136, 1) numpy object array of wikipedia page texts, used to fit wiki_model.

nips is a list holding one (7241, 1) numpy object array of NIPS conference paper texts (~181 MB download), used to fit nips_model.

wiki_model, nips_model, and sotus_model are sklearn Pipelines (CountVectorizer -> LatentDirichletAllocation, 50 topics) trained on the wiki, nips, and sotus corpora, respectively; each transforms text into 50-dimensional topic vectors. (The hosted files were pickled under an older scikit-learn; hypertools backfills newer estimator attributes on load so repr()/get_params()/transform() work under the installed version.)

The “shapes zoo” datasets – bunny, cube, dragon, sphere, teapot, vase, and biplane – are 3D point clouds of the corresponding objects (numpy arrays / DataFrames of x, y, z coordinates), useful for demonstrating alignment and plotting.

datasaurus is the “Datasaurus Dozen”: a list of 2D datasets with near-identical summary statistics but wildly different shapes.

reducestr, dict, class, instance, or fitted Reducer

Decomposition/manifold learning model to use (default: ‘IncrementalPCA’). Models supported: PCA, IncrementalPCA, SparsePCA, MiniBatchSparsePCA, KernelPCA, FastICA, FactorAnalysis, TruncatedSVD, DictionaryLearning, MiniBatchDictionaryLearning, TSNE, Isomap, SpectralEmbedding, LocallyLinearEmbedding, MDS, and UMAP; the mixture models GaussianMixture, BayesianGaussianMixture, LatentDirichletAllocation and NMF (GH #174); and the torch autoencoders Autoencoder, DeepAutoencoder, SparseAutoencoder, ConvolutionalAutoencoder, SequenceAutoencoder and VariationalAutoencoder (GH #162, pip install “hypertools[torch]”). Can be passed as a string, or for finer control as a dictionary, e.g. reduce={‘model’: ‘PCA’, ‘kwargs’: {‘whiten’: True}}. See scikit-learn model docs for details on parameters supported for each model.

ndimsint

Number of dimensions to reduce

alignstr, dict, False, or None

Alignment model to bring a list of datasets into a shared space. If str, ‘hyper’ (hyperalignment) or ‘SRM’ (shared response model). You can also pass a dictionary for finer control, where ‘model’ specifies the model and ‘kwargs’ holds its parameters, e.g. align={‘model’: ‘HyperAlign’, ‘kwargs’: {‘n_iter’: 10}}. If False or None, no alignment is applied (default: None).

normalizestr or False or None

If set to ‘across’, the columns of the input data will be z-scored across lists. That is, the z-scores will be computed with respect to column n across all arrays passed in the list. If set to ‘within’, the columns will be z-scored within each list that is passed. If set to ‘row’, each row of the input data will be z-scored. If set to False or None (default), the input data will be returned with no z-scoring.

legacybool

Pass legacy=True to read a dataset saved in the old pickled DataGeometry format by hypertools<0.8.0. Its raw data is extracted and returned like any other dataset – a DataGeometry object is never returned (that class is not part of the 1.0 API).

splitstring or None

Hugging Face datasets only: which split to load (default: the ‘train’ split if present, otherwise the first available split).

streamingbool

Hugging Face datasets only: if True, return a streaming IterableDataset instead of materializing the data (see https://huggingface.co/docs/datasets/en/stream). The result can be passed directly to hypertools.plot().

trustbool

Remote (non-built-in) sources only. Unpickling a payload fetched from a URL/Drive/Dropbox/Sheets executes arbitrary code embedded in it, so by default hypertools refuses to unpickle remote data (raising HypertoolsIOError – a warning is not a security boundary). Remote .npy/.npz payloads are likewise loaded with allow_pickle=False (raising if the array actually needs pickle support, e.g. an object array). Pass trust=True – only once you have verified the source – to allow unpickling remote data and pickle-backed remote arrays. This covers every remote-pickle path (extension-based, content-sniffed, and extensionless). Non-executable remote formats (.csv/.npz numeric/.parquet) never require trust. Built-in example datasets (listed below) are downloaded from a fixed, integrity-checked set of hosts and do not require this flag. Note that trust governs REMOTE sources; a local file path is treated as trusted and unpickled without it (matching numpy.load / pandas.read_pickle), so only hyp.load() a local pickle you created or trust. Local files are never subject to this policy.

Returns:
datanumpy array, DataFrame, list, or IterableDataset

The loaded raw data (a list of datasets when a list of strings was passed). If reduce/ndims/align/normalize are supplied, the analyzed data is returned directly.

Examples

>>> import hypertools
>>> hypertools.load('iris').columns.tolist()
['sepal length (cm)', 'sepal width (cm)', 'petal length (cm)',
 'petal width (cm)', 'target']
>>> hypertools.load('penguins').columns.tolist()
['species', 'island', 'bill_length_mm', 'bill_depth_mm',
 'flipper_length_mm', 'body_mass_g', 'sex']
>>> hypertools.load('fivethirtyeight/bechdel').shape  # 538's bechdel data
(1794, 15)
>>> hypertools.load('kaggle/uciml/iris').shape  # a Kaggle dataset
(150, 6)
>>> weights = hypertools.load('weights')  # built-in name always wins
>>> type(weights).__name__, len(weights)
('list', 36)

A list of strings resolves element-wise and returns a list of datasets that can be passed to any hypertools function.

Warning

Pickled payloads (.pkl/.geo) can execute arbitrary code when loaded – only load pickled data from sources you trust.