tokenDetails — Return a table of token details for tokenized documents.

tokenDetails(documents) returns a table describing the tokens stored in a tokenizedDocument object.

Syntax

tdetails = tokenDetails(documents)

Inputs

Returns

Errors

How tokenDetails works

• documents must be a tokenizedDocument object created by tokenizedDocument.
• Normally tokenized documents return table variables Token, DocumentNumber, LineNumber, Type, and Language.
• After addSentenceDetails, documents also return a SentenceNumber variable after DocumentNumber.
• After addPartOfSpeechDetails, documents also return a PartOfSpeech variable after Language.
• After addEntityDetails, documents also return an Entity variable after PartOfSpeech when part-of-speech details are present, otherwise after Language.
• After addLemmaDetails, documents also return a Lemma variable after Language; if multiple language details are present, PartOfSpeech precedes Entity, and Entity precedes Lemma.
• After addDependencyDetails, documents also return Head and Dependency variables after the other language-detail variables.
• Pre-tokenized documents created with TokenizeMethod, "none" initially return only Token and DocumentNumber, matching MATLAB's manually tokenized form.
• After addTypeDetails, pre-tokenized documents also return a Type variable.
• Type values are derived from RunMat's lightweight tokenizer and include letters, digits, punctuation, email-address, web-address, hashtag, at-mention, emoticon, emoji, and other.
• LineNumber is currently 1 for each token because RunMat's tokenizer stores document-level tokens and does not yet preserve per-line token provenance.
• Dependency, entity, and language annotations use the models available in the current Text Analytics implementation; their results can differ from other language-model implementations.
• DocumentNumber, SentenceNumber, LineNumber, and Head are generated double table columns. tokenDetails has no caller-provided native integer input.

GPU memory and residency

tokenDetails constructs a host table and has no provider kernel.

Examples

View Token Details

documents = tokenizedDocument(["alpha 123"; "visit https://example.com"]);
tdetails = tokenDetails(documents)

Expected output:

`tdetails` is a table with token text, document number, line number, token type, and language.

Add Types To Pre-Tokenized Documents

documents = tokenizedDocument(["For" "more" "information"], "TokenizeMethod", "none");
documents = addTypeDetails(documents);
tdetails = tokenDetails(documents)

Expected output:

`tdetails` includes `Token`, `DocumentNumber`, and `Type`.

Add Sentence Numbers

documents = tokenizedDocument("First sentence. Second sentence.");
documents = addSentenceDetails(documents);
tdetails = tokenDetails(documents)

Expected output:

`tdetails` includes `SentenceNumber`.

Add Lemmas

documents = tokenizedDocument("The dogs ran.");
documents = addLemmaDetails(documents);
tdetails = tokenDetails(documents)

Expected output:

`tdetails` includes `Lemma`.

Add Part-Of-Speech Tags

documents = tokenizedDocument("The dogs are running.");
documents = addPartOfSpeechDetails(documents);
tdetails = tokenDetails(documents)

Expected output:

`tdetails` includes `PartOfSpeech`.

Add Entity Tags

documents = tokenizedDocument("Mary uses MATLAB at MathWorks.");
documents = addEntityDetails(documents);
tdetails = tokenDetails(documents)

Expected output:

`tdetails` includes `Entity`.

Add Dependency Details

documents = tokenizedDocument("The dogs chase cats.");
documents = addDependencyDetails(documents);
tdetails = tokenDetails(documents)

Expected output:

`tdetails` includes `Head` and `Dependency`.

Using tokenDetails with coding agents

Open a RunMat example with live inputs, then ask the agent to explain how tokenDetails changes the result.

FAQ

Related Strings functions

Open-source implementation

Unlike proprietary runtimes, every RunMat function is open-source. Read exactly how tokenDetails is executed, line by line, in Rust.

• View the source for tokenDetails in Rust on GitHub
• Learn how the RunMat runtime works
• Found a bug? Open an issue with a minimal reproduction.

About RunMat

RunMat is an open-source runtime that executes MATLAB-syntax code blazing on any GPU. It is licensed under the Apache 2.0 license.

• RunMat automatically optimizes your math for GPU execution on Apple, Nvidia, and AMD hardware. No code changes needed. Simulations that took hours now take minutes.
• Start running code in seconds. RunMat runs in the browser, on the desktop, or from the CLI. No license server, no IT ticket.

Getting started · Benchmarks · Pricing