ChatGPT 3.5 gives very different result quality if used through OpenRouter against OpenAI

[sulth1]

Hello,

Again, thank you so much for this life-changing vault QA update.

I have been experimenting it today, and found that on your plugin, when I use ChatGPT 3.5 turbo 16k through Openrouter (https://openrouter.ai/models/openai/gpt-3.5-turbo-0125), the results are much better than through the GPT-3.5 16k option (not sure why it isn't labelled GPT-3.5 turbo 16k btw?).

Is there any reason for that? Aren't those two supposed to be the exact same? Could it have something to do with the temperature settings (kept as default 0.1)?

[logancyang]

Hi @sulth1, great observation! It's because OpenAI has updated gpt-3.5-turbo (GPT-3.5 in the plugin) to default to the new gpt-3.5-turbo-0125 which has 16K context window. The old GPT-3.5-TUBRO-16K is shown as legacy. We can remove it I guess.

See this from OpenAI:

[sulth1]

Interesting, thank you for the answer. Just to make sure I got it right:

• "GPT-3.5 16K" points to the 0613 version, aka that one on Openrouter : https://openrouter.ai/models/openai/gpt-3.5-turbo-0613
• "GPT-3.5" points to 0125, https://openrouter.ai/models/openai/gpt-3.5-turbo-0125 , who also has a 16k context

So using "GPT-3.5" or Openrouter with https://openrouter.ai/models/openai/gpt-3.5-turbo-0125 should give the exact same results.
Is that correct?

[logancyang]

"GPT-3.5 16K" points to the 0613 version

Yes, it is because gpt-3.5-turbo-16k is pointing to gpt-3.5-turbo-16k-0613 as the bottom of my screenshot shows.

aka that one on Openrouter : https://openrouter.ai/models/openai/gpt-3.5-turbo-0613

No, this is the gpt-3.5-turbo-0613 which has a 4096 context window.

"GPT-3.5" points to 0125, https://openrouter.ai/models/openai/gpt-3.5-turbo-0125 , who also has a 16k context

Yes

So using "GPT-3.5" or Openrouter with https://openrouter.ai/models/openai/gpt-3.5-turbo-0125 should give the exact same results.

Yes

In short, my gpt 3.5 and gpt 3.5 16k are using the model name gpt-3.5-turbo and gpt-3.5-turbo-16k behind the scenes. I don't include version numbers on purpose. These should be the latest versions of the corresponding model assuming OpenAI points them correctly on their end.