Max model output tokens?

[deniseiras]

I came across a 512 token cut in the response of the sabe-3 model.
Where can I check this information for all models?
How can I change the call to return more tokens?
Cheers!

[rodrigo-f-nogueira]

Hi Denis,

In the web chat, the maximum number of generated tokens is 4000.

In the API, you can configure it by setting the max_tokens parameters:

import maritalk

model = maritalk.MariTalk(
    key="insira sua chave aqui. Ex: '100088...'",
    model="sabia-3"
)

response = model.generate(
    "Quanto é 25 + 27?",
    max_tokens=200)

answer = response["answer"]

[deniseiras]

Hi,
thanks for your answer, it works.
What is the maximum value for max_tokens ? (At https://www.maritaca.ai/ I can see "200 mil tokens de saída por minuto". So it's possible to get completion with 200k tokens in a single call ? or is there limits related to message in HTTP call ?
Cheers

[rodrigo-f-nogueira]

Hi @deniseiras,

200k tokens/min is the rate limit, i.e., if you send requests such that more than 200k tokens are generated in one minute, the server will refuse to generate more tokens. Then you have to wait one minute or so to send new requests.

The max_tokens parameter is the amount of tokens generated in a single request. For instance, if you are using Sabia-3, which supports 32000 context tokens, and your prompt (input) has 1000 tokens, max_tokens can set to a maximum of 32000 - 1000 = 31000 tokens.

[deniseiras]

Thanks Rodrigo, I understand all now!

[rodrigo-f-nogueira]

Great, please let us if you any further questions