Is Tiktoken even needed/used?

[zackgilbert]

@krschacht As I was trying to work through some of the platform build issues, I noticed that the tiktoken_ruby doesn't actually appear to be used anywhere? Is this true? Should we remove it?

[krschacht]

@zackgilbert Hmm, I can't find it either. I used to take the user's message and estimate the number of tokens it was to make sure I wasn't going to exceed the token limit of the model. I was using tiktoken for that. But I just poked around in aibackend.rb and backend/openai.rb and neither of those seem to do that. It must have gotten removed at some point.

Feel free to remove tiktoken_ruby and assuming we don't get any failing tests it's probably safe to cut it. These days the token counts are getting so high on the models and they're only going to go up so it's far less necessary to check these things.