tokenizedDocument — Create tokenized document objects for Text Analytics workflows.

tokenizedDocument(str) tokenizes English or German text into a tokenizedDocument object. It also accepts pre-tokenized words with TokenizeMethod, "none", plus custom-token and regular-expression tokenization tables.

Syntax

documents = tokenizedDocument
documents = tokenizedDocument(str)
documents = tokenizedDocument(str, Name, Value, ...)

Inputs

Returns

Errors

How tokenizedDocument works

• tokenizedDocument with no input creates a scalar object containing one empty document.
• str may be a string scalar, string array, character vector, multi-row character array, or cell array of scalar text.
• String arrays and cell arrays preserve their document-array shape in the object's Shape property. Character matrices are treated as one document per row.
• tokenizedDocument(str, 'TokenizeMethod', 'unicode') tokenizes text into words and simple punctuation tokens.
• tokenizedDocument(words, 'TokenizeMethod', 'none') accepts a string array, character vector, or cell form as pre-tokenized words. A cell array of string arrays represents multiple pre-tokenized documents.
• Language currently accepts "en" or "de". Japanese, Korean, and MeCab tokenization are not implemented.
• DetectPatterns accepts "all", "none", or a string/cell vector containing one or more of "email-address", "web-address", "hashtag", "at-mention", and "emoticon". RunMat only keeps enabled complex patterns as single tokens.
• TopLevelDomains accepts a character vector, string array, or cell array of character vectors and controls web-address recognition when DetectPatterns includes "web-address". RunMat stores normalized TLD labels and uses custom lists for bare domains, www. addresses, and http:///https:// addresses; with the default list, explicit www. and scheme URLs are accepted when the host has a valid domain shape.
• CustomTokens accepts a character vector, string scalar/vector, cell text vector, or table with a required Token variable and optional Type variable. Custom tokens are kept as single tokens, conflicting matches use the longest token, and missing or empty table Type values default to "custom".
• RegularExpressions accepts a character vector, string scalar/vector, cell text vector, or table with a required Pattern variable and optional Type variable. Patterns are matched at the current tokenizer position, invalid or empty-matching patterns are rejected, conflicting regular-expression matches use the last matching pattern, and regular-expression tokens take precedence over custom tokens.
• The returned object stores Vocabulary, Documents, NumDocuments, DocumentLengths, Shape, TokenizeMethod, Language, DetectPatterns, and TopLevelDomains. Custom-token and regular-expression calls also store TypeDetails for tokenDetails.
• A document array is represented by one scalar object carrying document cells and Shape metadata. First-class object-array identity is not yet available for this class.
• Native numeric arrays are not document text or valid option values and are rejected before provider access. Document counts and lengths are stored as generated double metadata.

GPU memory and residency

tokenizedDocument tokenizes host-resident text and has no provider kernel.

Examples

Tokenize Text

documents = tokenizedDocument(["A short document"; "Another document"])

Expected output:

`documents.NumDocuments` is `2`, and `documents.Vocabulary` contains the unique tokens in first-seen order.

Use Pre-Tokenized Words

documents = tokenizedDocument(["alpha" "beta" "alpha"], "TokenizeMethod", "none")

Expected output:

`documents.DocumentLengths` is `3` for the single pre-tokenized document.

Choose Complex Token Patterns

documents = tokenizedDocument("Mail a@example.com #MATLAB at https://example.com", "DetectPatterns", ["email-address" "hashtag"])

Expected output:

The email address and hashtag stay as single tokens; the web address is tokenized as ordinary text and punctuation.

Configure Web Address Domains

documents = tokenizedDocument("See example.zz", "TopLevelDomains", "zz")

Expected output:

`example.zz` is recognized as a web-address token.

Use Custom Token Types

rules = table(["Na+"; "H2O"], ["ion"; "formula"], 'VariableNames', ["Token" "Type"])
documents = tokenizedDocument("Na+ in H2O", "CustomTokens", rules)

Expected output:

`Na+` and `H2O` stay as single tokens, and `tokenDetails(documents)` reports `ion` and `formula` type labels for those tokens.

Use Regular Expression Tokens

patterns = table(["[a-z]+\\d+"; "abc\\d+"], ["alnum"; "code"], 'VariableNames', ["Pattern" "Type"])
documents = tokenizedDocument("abc123 xyz9", "RegularExpressions", patterns)

Expected output:

`abc123` and `xyz9` stay as single tokens. The later matching `abc\\d+` pattern supplies the `code` type for `abc123`.

Using tokenizedDocument with coding agents

Open a RunMat example with live inputs, then ask the agent to explain how tokenizedDocument changes the result.

FAQ

Related Strings functions

Open-source implementation

Unlike proprietary runtimes, every RunMat function is open-source. Read exactly how tokenizedDocument is executed, line by line, in Rust.

• View the source for tokenizedDocument in Rust on GitHub
• Learn how the RunMat runtime works
• Found a bug? Open an issue with a minimal reproduction.

About RunMat

RunMat is an open-source runtime that executes MATLAB-syntax code blazing on any GPU. It is licensed under the Apache 2.0 license.

• RunMat automatically optimizes your math for GPU execution on Apple, Nvidia, and AMD hardware. No code changes needed. Simulations that took hours now take minutes.
• Start running code in seconds. RunMat runs in the browser, on the desktop, or from the CLI. No license server, no IT ticket.

Getting started · Benchmarks · Pricing