Type Alias TokenizationResult

Tokenization output with a normalized string and its tokens.

type TokenizationResult = {
normalized: string;
tokens: string[];
}

Index

Properties

normalized tokens

Properties

normalized

normalized: string

Normalized input string used to produce tokens.

tokens

tokens: string[]

Token list derived from the normalized input.