addTypeDetails — Add token type details to tokenized documents.
addTypeDetails(documents) detects token types in a RunMat tokenizedDocument compatibility object and returns an updated object. addTypeDetails(documents,'DiscardKnownValues',tf) controls whether existing type details are preserved or recomputed. addTypeDetails(documents,'TopLevelDomains',domains) customizes web-address type recognition.
Syntax
newDocuments = addTypeDetails(documents)
newDocuments = addTypeDetails(documents,Name,Value)Inputs
| Name | Type | Required | Default | Description |
|---|---|---|---|---|
documents | Any | Yes | — | tokenizedDocument object. |
NameValue | Any | Variadic | — | Name-value options: DiscardKnownValues, TopLevelDomains. |
Returns
| Name | Type | Description |
|---|---|---|
newDocuments | Any | Updated tokenized document object. |
Errors
| Identifier | When | Message |
|---|---|---|
RunMat:addTypeDetails:InvalidInput | Input is not a supported tokenizedDocument object or option form. | addTypeDetails: invalid input |
How addTypeDetails works
documentsmust be a RunMattokenizedDocumentobject.- When type details are absent, RunMat derives one type value for each token and stores the details on the returned object.
DiscardKnownValuesdefaults tofalse, so existing known type details are preserved while missing orunknowntype details are filled.DiscardKnownValues,truerecomputes every stored type detail.TopLevelDomainsaccepts a character vector, string array, or cell array of character vectors. It overrides the document's stored TLD list for web-address type recognition and is stored on the returned object.- RunMat recognizes
letters,digits,punctuation,email-address,web-address,hashtag,at-mention,emoticon,emoji, andotherusing the same lightweight token rules astokenizedDocument. - Custom token-type tables, full MathWorks
topLevelDomainscorpus parity, and exact MathWorks URL classifier parity remain broader Text Analytics work.
GPU memory and residency
addTypeDetails updates host tokenizedDocument metadata and has no provider kernel.
Examples
Add Types To Pre-Tokenized Documents
documents = tokenizedDocument(["For" "more" "information" ","], "TokenizeMethod", "none");
documents = addTypeDetails(documents);
tdetails = tokenDetails(documents)Expected output:
`tdetails.Type` contains `letters`, `letters`, `letters`, and `punctuation`.Recompute Type Details
documents = addTypeDetails(documents, "DiscardKnownValues", true)Expected output:
`documents` has refreshed token type details.Configure Web Address Domains
documents = tokenizedDocument(["example.zz"], "TokenizeMethod", "none");
documents = addTypeDetails(documents, "TopLevelDomains", "zz");
tdetails = tokenDetails(documents)Expected output:
`tdetails.Type` contains `web-address` for `example.zz`.Using addTypeDetails with coding agents
Open a RunMat example with live inputs, then ask the agent to explain how addTypeDetails changes the result.
Run a small addTypeDetails example, explain the result, then change one input and compare the output.
FAQ
Does addTypeDetails support TopLevelDomains?⌄
Yes. RunMat accepts character, string-array, and cell-string TLD lists for web-address detection. Full default topLevelDomains corpus parity remains tracked.
Does addTypeDetails execute on the GPU?⌄
No. It classifies host token strings and updates object metadata.
Related Strings functions
Text Analytics
addDependencyDetails · addEntityDetails · addLemmaDetails · addPartOfSpeechDetails · addSentenceDetails · bagOfNgrams · bagOfWords · cosineSimilarity · doc2sequence · encode · extractFileText · extractHTMLText · fastTextWordEmbedding · findElement · getAttribute · htmlTree · ind2word · isVocabularyWord · normalizeWords · readWordEmbedding · removeLongWords · removeShortWords · removeStopWords · removeWords · stopWords · tokenDetails · tokenizedDocument · trainWordEmbedding · vaderSentimentScores · vec2word · word2ind · word2vec · wordEncoding · writeWordEmbedding
Transform
append · deblank · erase · eraseBetween · erasePunctuation · eraseURLs · extractAfter · extractBefore · extractBetween · insertAfter · insertBefore · join · lower · pad · replace · replaceBetween · reverse · split · splitlines · strcat · strip · strjoin · strjust · strrep · strsplit · strtrim · upper
Core
blanks · char · compose · convertCharsToStrings · convertContainedStringsToChars · convertStringsToChars · genvarname · int2str · isletter · isspace · isStringScalar · isstrprop · mat2str · native2unicode · newline · num2str · sprintf · sscanf · str2double · str2num · strcmp · strcmpi · string · string.empty · strings · strlength · strncmp · strncmpi · strtok · unicode2native
Search
contains · endsWith · matches · startsWith · strfind
Pattern
digitsPattern · lettersPattern · pattern · regexpPattern · textBoundary · wildcardPattern
Open-source implementation
Unlike proprietary runtimes, every RunMat function is open-source. Read exactly how addTypeDetails is executed, line by line, in Rust.
- View the source for addTypeDetails in Rust on GitHub
- Learn how the RunMat runtime works
- Found a bug? Open an issue with a minimal reproduction.
About RunMat
RunMat is an open-source runtime that executes MATLAB-syntax code blazing on any GPU. It is licensed under the Apache 2.0 license.
- RunMat automatically optimizes your math for GPU execution on Apple, Nvidia, and AMD hardware. No code changes needed. Simulations that took hours now take minutes.
- Start running code in seconds. RunMat runs in the browser, on the desktop, or from the CLI. No license server, no IT ticket.