hypertools.tools.text2mat

hypertools.tools.text2mat(data, vectorizer='CountVectorizer', semantic='LatentDirichletAllocation', corpus='wiki')

Turns a list of text samples into a matrix using a vectorizer and a text model

Parameters:

datalist (or list of lists) of text samples

The text data to transform

vectorizerstr, dict, class or class instance

The vectorizer to use. A string name is resolved in three tiers (GH #198): (1) the built-in scikit-learn registry – ‘CountVectorizer’ or ‘TfidfVectorizer’; (2) if not found there, the gensim wrapper registry (hypertools.tools.gensim_models, requires pip install “hypertools[gensim]”) – ‘Word2Vec’, ‘Doc2Vec’, or ‘FastText’; (3) if still not found, the name is treated as a Hugging Face sentence-transformers model name/id (e.g. ‘all-MiniLM-L6-v2’), via data-wrangler’s HF embedding support. To change default parameters, set to a dictionary e.g. {‘model’ : ‘CountVectorizer’, ‘kwargs’ : {‘max_features’ : 10}} (the legacy {‘model’, ‘params’} form is also still accepted). See https://scikit-learn.org/stable/api/sklearn.feature_extraction.html for scikit-learn details. You can also specify your own vectorizer model as a class, or class instance. With either option, the class must have a fit_transform method (see https://scikit-learn.org/stable/data_transforms.html). To set parameters, use the dict form (or a configured class instance); a bare class is instantiated with its defaults.

semanticstr, dict, class or class instance

Text model to use to transform text data. Built-in options are ‘LatentDirichletAllocation’ or ‘NMF’ (default: LDA). String names are resolved using the same three-tier order as vectorizer (sklearn -> gensim -> HuggingFace, GH #198); the gensim tier adds ‘LdaModel’, ‘LsiModel’, and ‘HdpModel’. To change default parameters, set to a dictionary e.g. {‘model’ : ‘NMF’, ‘kwargs’ : {‘n_components’ : 10}} (the legacy {‘model’, ‘params’} form is also still accepted). See https://scikit-learn.org/stable/api/sklearn.decomposition.html for details on the two scikit-learn model options. You can also specify your own text model as a class, or class instance. With either option, the class must have a fit_transform method (see https://scikit-learn.org/stable/data_transforms.html). To set parameters, use the dict form (or a configured class instance); a bare class is instantiated with its defaults.

corpuslist (or list of lists) of text samples or ‘wiki’, ‘nips’, ‘sotus’.

Text to use to fit the semantic model (optional). If set to ‘wiki’, ‘nips’ or ‘sotus’ and the default semantic and vectorizer models are used, a pretrained model will be loaded which can save a lot of time.

Returns:

transformed datalist of numpy arrays

The transformed text data