docs/concepts/tokens/

[giscus[bot]]

docs/concepts/tokens/

Modern large language models (LLMs) are typically based on a transformer architecture that processes a sequence of units known as tokens. Tokens are the fundamental elements that models use to break down input and generate output. In this section, we'll discuss what tokens are and how they are used by language models.

https://python.langchain.com/docs/concepts/tokens/

[newkdr]

H

[shan-max]

Yes this is reallity.