addTypeDetails — Detect token types in tokenized documents.
addTypeDetails adds token-type metadata to a RunMat tokenizedDocument object and can override top-level domains used for web-address recognition.
Syntax
newDocuments = addTypeDetails(documents)
newDocuments = addTypeDetails(documents,Name,Value)Inputs
documents- Type
Any- Required
- Yes
- Description
- tokenizedDocument object.
NameValue- Type
Any- Required
- Variadic
- Description
- Name-value options: DiscardKnownValues, TopLevelDomains.
Returns
newDocuments- Type
Any- Description
- Updated tokenized document object.
Errors
RunMat:addTypeDetails:InvalidInput- When
- Input is not a supported tokenizedDocument object or option form.
- Message
addTypeDetails: invalid input
Supported forms
updatedDocuments = addTypeDetails(documents)
updatedDocuments = addTypeDetails(documents, 'DiscardKnownValues', tf)
updatedDocuments = addTypeDetails(documents, 'TopLevelDomains', domains)
updatedDocuments is a tokenizedDocument compatibility object.
documents must be a RunMat tokenizedDocument object.
tf must be a logical scalar; numeric and integer 0/1 values are rejected.
domains must contain one or more valid top-level-domain labels.
Use tokenDetails(updatedDocuments) to view the resulting type table.
Token types
addTypeDetails(documents) detects token types in a RunMat tokenizedDocument object. If details are absent, one type is derived for each token. DiscardKnownValues defaults to logical false, preserving known types and filling missing or unknown values; logical true recomputes every type.
TopLevelDomains accepts a character vector, string array, or cell array of character vectors. It overrides the document's stored TLD list for web-address recognition and is stored on the returned object.
The lightweight classifier recognizes letters, digits, punctuation, email-address, web-address, hashtag, at-mention, emoticon, emoji, and other. Custom token-type tables, full MathWorks topLevelDomains corpus parity, and exact URL-classifier parity remain open.
Execution
The builtin classifies host token strings and updates object metadata; it has no provider kernel or fusion path.
Examples
Add types to pre-tokenized documents
documents = tokenizedDocument(["For" "more" "information" ","], "TokenizeMethod", "none"); updated = addTypeDetails(documents); tdetails = tokenDetails(updated)Expected output:
tdetails.Type includes letters and punctuationRecompute known types
documents = tokenizedDocument("For more information"); updated = addTypeDetails(documents, "DiscardKnownValues", true)Expected output:
updated has refreshed token type detailsConfigure web-address domains
documents = tokenizedDocument("example.zz", "TokenizeMethod", "none"); updated = addTypeDetails(documents, "TopLevelDomains", "zz"); tdetails = tokenDetails(updated)Expected output:
tdetails.Type contains web-addressUsing addTypeDetails with coding agents
Open a RunMat example with live inputs, then ask the agent to explain how addTypeDetails changes the result.
Run a small addTypeDetails example, explain the result, then change one input and compare the output.
FAQ
Does addTypeDetails support TopLevelDomains?⌄
Yes. Character, string-array, and cell-string lists customize web-address detection; full default corpus parity remains open.
Does addTypeDetails execute on the GPU?⌄
No. It classifies host token strings and updates metadata.
Related Strings functions
Text Analytics
addDependencyDetails · addEntityDetails · addLemmaDetails · addPartOfSpeechDetails · addSentenceDetails · bagOfNgrams · bagOfWords · cosineSimilarity · doc2sequence · encode · extractFileText · extractHTMLText · fastTextWordEmbedding · findElement · getAttribute · htmlTree · ind2word · isVocabularyWord · normalizeWords · readWordEmbedding · removeLongWords · removeShortWords · removeStopWords · removeWords · stopWords · tokenDetails · tokenizedDocument · trainWordEmbedding · vaderSentimentScores · vec2word · word2ind · word2vec · wordEncoding · writeWordEmbedding
Transform
append · deblank · erase · eraseBetween · erasePunctuation · eraseURLs · extractAfter · extractBefore · extractBetween · insertAfter · insertBefore · join · lower · pad · replace · replaceBetween · reverse · split · splitlines · strcat · strip · strjoin · strjust · strrep · strsplit · strtrim · upper
Core
blanks · char · compose · convertCharsToStrings · convertContainedStringsToChars · convertStringsToChars · genvarname · int2str · isletter · isspace · isStringScalar · isstrprop · mat2str · native2unicode · newline · num2str · sprintf · sscanf · str2double · str2num · strcmp · strcmpi · string · string.empty · strings · strlength · strncmp · strncmpi · strtok · unicode2native
Search
contains · endsWith · matches · startsWith · strfind
Pattern
digitsPattern · lettersPattern · pattern · regexpPattern · textBoundary · wildcardPattern
Source you can read
Every RunMat function's source is public. Read exactly how addTypeDetails is executed, line by line, in Rust.
- View the source for addTypeDetails in Rust on GitHub
- Learn how the RunMat runtime works
- Found a bug? Open an issue with a minimal reproduction.
About RunMat
RunMat is a runtime that executes MATLAB-syntax code fast on any GPU. The runtime is free for individuals, including freelancers; students and faculty at an academic institution, for teaching or research there; contributors to an open-source project, for work on that project; organizations with 250 or fewer people and US$10M or less in annual revenue; and anyone, for non-production use. See runmat.com/license.
- RunMat automatically optimizes your math for GPU execution on Apple, Nvidia, and AMD hardware. No code changes needed. Simulations that took hours now take minutes.
- Start running code in seconds. RunMat runs in the browser, on the desktop, or from the CLI. No license server, no IT ticket.