CodeTokenizer

Functional interface converting source code text into colored syntax tokens.

Overview

Implementations of CodeTokenizer scan editor text in a CodeTokenizationContext and emit a list of CodeToken ranges for syntax rendering.

API

CodeTokenizer::tokenize(CodeTokenizationContext)

Tokenize Text

Parses text into list of styled syntax tokens.

Parameter	Type	Description
context	CodeTokenizationContext	—

Returns List.