Function tokenize

tokenize(input: string): TokenizationResult
Normalize text and split into word tokens for exact/fulltext indexing.
Parameters
- input: string
  Raw input text to normalize and tokenize.
Returns TokenizationResult
Normalized string and token list.
- Defined in api/Indexing/tokenize.ts:19