normalizeWords — Stem or lemmatize words and tokenized documents.

normalizeWords(words) reduces each standalone word to a root form. normalizeWords(documents) updates RunMat tokenizedDocument compatibility objects while preserving document shape and metadata.

Syntax

updatedDocuments = normalizeWords(documents)
updatedWords = normalizeWords(words)
updatedWords = normalizeWords(words, Name, Value, ...)

Inputs

Returns

Errors

How normalizeWords works

• normalizeWords(words) uses English stemming by default.
• normalizeWords(words, 'Language', language) accepts "en" for English or "de" for German on standalone word input.
• normalizeWords(___, 'Style', style) accepts "stem" or "lemma"; English lemmatization uses RunMat's lightweight compatibility rule set for standalone words and tokenized documents.
• normalizeWords(documents) normalizes tokens whose derived token type is letters or other, and preserves punctuation, URLs, hashtags, mentions, and emoticons.
• For tokenizedDocument input, RunMat uses the object's Language metadata. The Language name-value option is only accepted for standalone word input.
• String arrays return string arrays with the same shape.
• Character vectors and character arrays return character arrays; multi-row character arrays are treated as one word per row and padded after normalization.
• Cell arrays preserve the cell shape and preserve scalar string versus character-vector element types.
• Each text element must contain a single word. Multi-word elements raise RunMat:normalizeWords:InvalidInput.
• Missing string values are preserved.
• RunMat supports lightweight English and German tokenizedDocument normalization. Japanese/Korean MeCab-backed token details, full dictionary-backed and part-of-speech-aware lemmatization, and exact MathWorks dictionary identity remain tracked by the broader Text Analytics compatibility issue.
• English stemming follows a Porter-style algorithm. German stemming is conservative and intended for compatibility with common preprocessing workflows, not exact release-to-release MathWorks dictionary identity.

GPU memory and residency

normalizeWords operates on host-resident text and has no provider kernel.

Examples

Stem Words In A String Array

words = ["a" "strongly" "worded" "collection" "of" "words"];
newWords = normalizeWords(words)

Expected output:

newWords = 1x6 string
    "a"    "strongli"    "word"    "collect"    "of"    "word"

Lemmatize English Words

words = ["building" "has" "floors"];
newWords = normalizeWords(words, "Style", "lemma")

Expected output:

newWords = 1x3 string
    "build"    "have"    "floor"

Stem German Words

words = ["Morgen"; "guter"];
newWords = normalizeWords(words, "Language", "de")

Expected output:

newWords = 2x1 string
    "morg"
    "gut"

Stem Tokenized Documents

documents = tokenizedDocument("a strongly worded collection of words");
newDocuments = normalizeWords(documents)

Expected output:

`newDocuments` is a tokenizedDocument containing `a strongli word collect of word`.

Using normalizeWords with coding agents

Open a RunMat example with live inputs, then ask the agent to explain how normalizeWords changes the result.

FAQ

Related Strings functions

Open-source implementation

Unlike proprietary runtimes, every RunMat function is open-source. Read exactly how normalizeWords is executed, line by line, in Rust.

• View the source for normalizeWords in Rust on GitHub
• Learn how the RunMat runtime works
• Found a bug? Open an issue with a minimal reproduction.

About RunMat

RunMat is an open-source runtime that executes MATLAB-syntax code blazing on any GPU. It is licensed under the Apache 2.0 license.

• RunMat automatically optimizes your math for GPU execution on Apple, Nvidia, and AMD hardware. No code changes needed. Simulations that took hours now take minutes.
• Start running code in seconds. RunMat runs in the browser, on the desktop, or from the CLI. No license server, no IT ticket.

Getting started · Benchmarks · Pricing