Function highlight 

pub fn highlight(
    language: Language,
    text: &str,
    palette: &Palette,
) -> Vec<TextRun>

Tokenize text into runs covering every byte: claimed tokens in their colors, the gaps between them in the palette’s base style. Byte offsets throughout, which is what the document indexes by, and why a multi-byte character in a comment cannot shift the styling of what follows it.