Tokenizer

The component that splits text into tokens and maps them to integer ids, and back. Each model family has its own, so count tokens with the tokenizer of the model you call.

All terms, A to Z