tokenizedDocument — Create tokenized document objects for Text Analytics workflows.

tokenizedDocument(str) tokenizes text into a lightweight RunMat tokenizedDocument object. This slice supports English and German Unicode-style tokenization, pre-tokenized word input through TokenizeMethod, "none", and documented custom-token and regular-expression tokenization tables.

Syntax

documents = tokenizedDocument
documents = tokenizedDocument(str)
documents = tokenizedDocument(str, Name, Value, ...)

Inputs

Returns

Errors

How tokenizedDocument works

• tokenizedDocument with no input creates a scalar tokenizedDocument compatibility object containing one empty document.
• str may be a string scalar, string array, character vector, multi-row character array, or cell array of scalar text.
• String arrays and cell arrays preserve their document-array shape in the object's Shape property. Character matrices are treated as one document per row.
• tokenizedDocument(str, 'TokenizeMethod', 'unicode') tokenizes text into words and simple punctuation tokens.
• tokenizedDocument(words, 'TokenizeMethod', 'none') accepts a string array, character vector, or cell form as pre-tokenized words. A cell array of string arrays represents multiple pre-tokenized documents.
• Language accepts "en" or "de". Japanese and Korean require MeCab-compatible tokenization and remain tracked by the Text Analytics umbrella issue.
• DetectPatterns accepts "all", "none", or a string/cell vector containing one or more of "email-address", "web-address", "hashtag", "at-mention", and "emoticon". RunMat only keeps enabled complex patterns as single tokens.
• TopLevelDomains accepts a character vector, string array, or cell array of character vectors and controls web-address recognition when DetectPatterns includes "web-address". RunMat stores normalized TLD labels and uses custom lists for bare domains, www. addresses, and http:///https:// addresses; with the default list, explicit www. and scheme URLs are accepted when the host has a valid domain shape.
• CustomTokens accepts a character vector, string scalar/vector, cell text vector, or table with a required Token variable and optional Type variable. Custom tokens are kept as single tokens, conflicting matches use the longest token, and missing or empty table Type values default to "custom".
• RegularExpressions accepts a character vector, string scalar/vector, cell text vector, or table with a required Pattern variable and optional Type variable. Patterns are matched at the current tokenizer position, invalid or empty-matching patterns are rejected, conflicting regular-expression matches use the last matching pattern, and regular-expression tokens take precedence over custom tokens.
• The returned object stores Vocabulary plus RunMat compatibility metadata properties: Documents, NumDocuments, DocumentLengths, Shape, TokenizeMethod, Language, DetectPatterns, and TopLevelDomains. When CustomTokens or RegularExpressions are used, RunMat also stores TypeDetails so tokenDetails returns the custom type labels. tokenDetails derives common token type and language columns from metadata; pre-tokenized TokenizeMethod, "none" documents need addTypeDetails before a Type column is returned.
• RunMat represents a MATLAB tokenizedDocument array as one scalar object carrying document cells and shape metadata because generic first-class MATLAB object arrays are not yet available for this class.
• Full MathWorks topLevelDomains corpus parity, exact ICU/MeCab tokenization, exact statistical sentence/part-of-speech/entity/dependency model parity, and object-array identity parity remain tracked in the broader Text Analytics compatibility issue.

GPU memory and residency

tokenizedDocument tokenizes host-resident text and has no provider kernel.

Examples

Tokenize Text

documents = tokenizedDocument(["A short document"; "Another document"])

Expected output:

`documents.NumDocuments` is `2`, and `documents.Vocabulary` contains the unique tokens in first-seen order.

Use Pre-Tokenized Words

documents = tokenizedDocument(["alpha" "beta" "alpha"], "TokenizeMethod", "none")

Expected output:

`documents.DocumentLengths` is `3` for the single pre-tokenized document.

Choose Complex Token Patterns

documents = tokenizedDocument("Mail a@example.com #MATLAB at https://example.com", "DetectPatterns", ["email-address" "hashtag"])

Expected output:

The email address and hashtag stay as single tokens; the web address is tokenized as ordinary text and punctuation.

Configure Web Address Domains

documents = tokenizedDocument("See example.zz", "TopLevelDomains", "zz")

Expected output:

`example.zz` is recognized as a web-address token.

Use Custom Token Types

rules = table(["Na+"; "H2O"], ["ion"; "formula"], 'VariableNames', ["Token" "Type"])
documents = tokenizedDocument("Na+ in H2O", "CustomTokens", rules)

Expected output:

`Na+` and `H2O` stay as single tokens, and `tokenDetails(documents)` reports `ion` and `formula` type labels for those tokens.

Use Regular Expression Tokens

patterns = table(["[a-z]+\\d+"; "abc\\d+"], ["alnum"; "code"], 'VariableNames', ["Pattern" "Type"])
documents = tokenizedDocument("abc123 xyz9", "RegularExpressions", patterns)

Expected output:

`abc123` and `xyz9` stay as single tokens. The later matching `abc\\d+` pattern supplies the `code` type for `abc123`.

Using tokenizedDocument with coding agents

Open a RunMat example with live inputs, then ask the agent to explain how tokenizedDocument changes the result.

FAQ

Related Strings functions

Open-source implementation

Unlike proprietary runtimes, every RunMat function is open-source. Read exactly how tokenizedDocument is executed, line by line, in Rust.

• View the source for tokenizedDocument in Rust on GitHub
• Learn how the RunMat runtime works
• Found a bug? Open an issue with a minimal reproduction.

About RunMat

RunMat is an open-source runtime that executes MATLAB-syntax code blazing on any GPU. It is licensed under the Apache 2.0 license.

• RunMat automatically optimizes your math for GPU execution on Apple, Nvidia, and AMD hardware. No code changes needed. Simulations that took hours now take minutes.
• Start running code in seconds. RunMat runs in the browser, on the desktop, or from the CLI. No license server, no IT ticket.

Getting started · Benchmarks · Pricing