hypertools.normalize

hypertools.normalize(x, normalize='across', internal=False, format_data=True, impute=None, return_model=False, manip=None, reduce=None, ndims=None, align=None, cluster=None, model=None)

Z-transform the columns or rows of an array, or list of arrays

This function normalizes the rows or columns of the input array(s). This can be useful because data reduction and machine learning techniques are sensitive to scaling differences between features. By default, the function is set to normalize ‘across’ the columns of all lists, but it can also normalize the columns ‘within’ each individual list, or alternatively, for each row in the array.

Parameters:

xNumpy array or list of arrays

This can either be a single array, or list of arrays

normalizestr, False, None, or fitted Normalizer

If set to ‘across’, the columns of the input data will be z-scored across lists (default). That is, the z-scores will be computed with respect to column n across all arrays passed in the list. If set to ‘within’, the columns will be z-scored within each list that is passed. If set to ‘row’, each row of the input data will be z-scored. If set to False or None, the input data will be returned with no z-scoring. A previously-fitted Normalizer (as returned by return_model=True) is applied via .transform instead of being refit. Any other value raises a ValueError.

internalbool

If True, ALWAYS return a list (one array per input dataset), even for single-dataset input – used by hypertools’ own pipeline plumbing (default: False).

format_databool

Whether or not to first call the format_data function (default: True).

imputestr, dict, class, class instance or None

Overrides the default PPCA missing-data fill applied during the format_data stage with a different hypertools.impute model (default: None, i.e. PPCA). Only used when format_data is True.

return_modelbool

If True, also return the fitted model: the fitted Normalizer when only the normalize stage ran, or a fitted hypertools.Pipeline when manip=/reduce=/align=/cluster= made multiple stages run (default: False).

manip, reduce, align, clustermodel spec or None

Cross-module stage kwargs (GH #138): when any of these is given, the other stages also run (via hypertools.core.pipeline.build_pipeline), in the canonical order manip -> normalize -> reduce -> align -> cluster (GH #153), with this function’s own normalize= slotted in at the normalize stage (default: None for all four, i.e. only normalize runs).

ndimsint or None

Passed through to the reduce stage (as ndims=) when reduce= is also given.

modelsame forms as normalize, or None

Alias for normalize=, so the own-stage spec can be spelled model= here exactly as in hyp.manip/hyp.impute/hyp.predict/ hyp.align (release-1.0 audit: the sibling APIs used two different kwarg conventions). Pass only one of normalize=/model=; passing both (with different values) raises ValueError (default: None).

Returns:

normalized_xNumpy array or list of arrays

An array or list of arrays where the columns or rows are z-scored. If the input was a list with more than one element, a list is returned; a single array – or a single-element list – returns a bare array. DataFrame inputs are converted to arrays (index/column metadata is not preserved; use hypertools.manip with model='ZScore' to keep it). If return_model=True, a (normalized_x, model) tuple is returned instead.

Notes

Standard deviations are POPULATION standard deviations (ddof=0, the np.std/scipy.stats.zscore convention). hypertools.manip’s ZScore manipulator uses the SAMPLE standard deviation (ddof=1, the pandas convention), so the two z-scoring entry points differ by a factor of sqrt(n / (n - 1)). Missing values (NaN) are PPCA-imputed during the format_data stage (when format_data=True); hypertools.manip propagates NaNs unchanged.

Examples

>>> import numpy as np
>>> from hypertools import normalize
>>> x = np.array([[1., 0.], [2., 10.], [3., 20.], [4., 30.]])
>>> z = normalize(x, normalize='within')
>>> np.allclose(z.mean(axis=0), 0.0) and np.allclose(z.std(axis=0), 1.0)
True
>>> a, b = np.zeros((5, 2)), np.ones((5, 2))
>>> z_across = normalize([a, b], normalize='across')
>>> np.allclose(np.vstack(z_across).mean(axis=0), 0.0)
True