normalizeWords — Stem or lemmatize words and tokenized documents.
normalizeWords(words) reduces each standalone word to a root form. normalizeWords(documents) updates RunMat tokenizedDocument compatibility objects while preserving document shape and metadata.
Syntax
updatedDocuments = normalizeWords(documents)
updatedWords = normalizeWords(words)
updatedWords = normalizeWords(words, Name, Value, ...)Inputs
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
words | Any | Yes | — | Words to stem or lemmatize. |
NameValue | Any | Variadic | — | Name-value options: Language and Style. |
Returns
| Name | Type | Description |
|---|---|---|
updatedWords | Any | Normalized words or tokenizedDocument object. |
Errors
| Identifier | When | Message |
|---|---|---|
RunMat:normalizeWords:InvalidInput | Inputs are not a supported normalizeWords form. | normalizeWords: invalid input |
How normalizeWords works
normalizeWords(words)uses English stemming by default.normalizeWords(words, 'Language', language)accepts"en"for English or"de"for German on standalone word input.normalizeWords(___, 'Style', style)accepts"stem"or"lemma"; English lemmatization uses RunMat's lightweight compatibility rule set for standalone words and tokenized documents.normalizeWords(documents)normalizes tokens whose derived token type islettersorother, and preserves punctuation, URLs, hashtags, mentions, and emoticons.- For
tokenizedDocumentinput, RunMat uses the object'sLanguagemetadata. TheLanguagename-value option is only accepted for standalone word input. - String arrays return string arrays with the same shape.
- Character vectors and character arrays return character arrays; multi-row character arrays are treated as one word per row and padded after normalization.
- Cell arrays preserve the cell shape and preserve scalar string versus character-vector element types.
- Each text element must contain a single word. Multi-word elements raise
RunMat:normalizeWords:InvalidInput. - Missing string values are preserved.
- RunMat supports lightweight English and German
tokenizedDocumentnormalization. Japanese/Korean MeCab-backed token details, full dictionary-backed and part-of-speech-aware lemmatization, and exact MathWorks dictionary identity remain tracked by the broader Text Analytics compatibility issue. - English stemming follows a Porter-style algorithm. German stemming is conservative and intended for compatibility with common preprocessing workflows, not exact release-to-release MathWorks dictionary identity.
- Integer semantics:
normalizeWordshas no numeric input role. Numeric, logical, symbolic, and provider-resident values are rejected before gather or provider access.
GPU memory and residency
normalizeWords operates on host-resident text and has no provider kernel.
Examples
Stem Words In A String Array
words = ["a" "strongly" "worded" "collection" "of" "words"];
newWords = normalizeWords(words)Expected output:
newWords = 1x6 string
"a" "strongli" "word" "collect" "of" "word"Lemmatize English Words
words = ["building" "has" "floors"];
newWords = normalizeWords(words, "Style", "lemma")Expected output:
newWords = 1x3 string
"build" "have" "floor"Stem German Words
words = ["Morgen"; "guter"];
newWords = normalizeWords(words, "Language", "de")Expected output:
newWords = 2x1 string
"morg"
"gut"Stem Tokenized Documents
documents = tokenizedDocument("a strongly worded collection of words");
newDocuments = normalizeWords(documents)Expected output:
`newDocuments` is a tokenizedDocument containing `a strongli word collect of word`.Using normalizeWords with coding agents
Open a RunMat example with live inputs, then ask the agent to explain how normalizeWords changes the result.
Run a small normalizeWords example, explain the result, then change one input and compare the output.
FAQ
Does normalizeWords support tokenizedDocument input?⌄
Yes, for RunMat's lightweight English and German tokenizedDocument compatibility objects. It preserves complex tokens and punctuation and normalizes derived letters and other tokens.
Can standalone words use Japanese or Korean?⌄
No. RunMat rejects Japanese and Korean standalone word normalization with an error that points to the remaining tokenizedDocument work.
Does normalizeWords execute on the GPU?⌄
No. Word normalization operates on host strings and has no runmat-accelerate provider path.
Related Strings functions
Text Analytics
addDependencyDetails · addEntityDetails · addLemmaDetails · addPartOfSpeechDetails · addSentenceDetails · addTypeDetails · bagOfNgrams · bagOfWords · cosineSimilarity · doc2sequence · encode · extractFileText · extractHTMLText · fastTextWordEmbedding · findElement · getAttribute · htmlTree · ind2word · isVocabularyWord · readWordEmbedding · removeLongWords · removeShortWords · removeStopWords · removeWords · stopWords · tokenDetails · tokenizedDocument · trainWordEmbedding · vaderSentimentScores · vec2word · word2ind · word2vec · wordEncoding · writeWordEmbedding
Transform
append · deblank · erase · eraseBetween · erasePunctuation · eraseURLs · extractAfter · extractBefore · extractBetween · insertAfter · insertBefore · join · lower · pad · replace · replaceBetween · reverse · split · splitlines · strcat · strip · strjoin · strjust · strrep · strsplit · strtrim · upper
Core
blanks · char · compose · convertCharsToStrings · convertContainedStringsToChars · convertStringsToChars · genvarname · int2str · isletter · isspace · isStringScalar · isstrprop · mat2str · native2unicode · newline · num2str · sprintf · sscanf · str2double · str2num · strcmp · strcmpi · string · string.empty · strings · strlength · strncmp · strncmpi · strtok · unicode2native
Search
contains · endsWith · matches · startsWith · strfind
Pattern
digitsPattern · lettersPattern · pattern · regexpPattern · textBoundary · wildcardPattern
Open-source implementation
Unlike proprietary runtimes, every RunMat function is open-source. Read exactly how normalizeWords is executed, line by line, in Rust.
- View the source for normalizeWords in Rust on GitHub
- Learn how the RunMat runtime works
- Found a bug? Open an issue with a minimal reproduction.
About RunMat
RunMat is an open-source runtime that executes MATLAB-syntax code blazing on any GPU. It is licensed under the Apache 2.0 license.
- RunMat automatically optimizes your math for GPU execution on Apple, Nvidia, and AMD hardware. No code changes needed. Simulations that took hours now take minutes.
- Start running code in seconds. RunMat runs in the browser, on the desktop, or from the CLI. No license server, no IT ticket.