openai: add option to embed documents without tokenizing

[efriis]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Like #7815, add the ability to bypass tokenization in embed_documents in openai embeddings

Motivation

To be discussed

Proposal (If applicable)

Something along the lines of #7815

[efriis]

Hey @sudeepag! Could you clarify what the goal of #7815 is?

I'm curious if tiktoken_enabled=False solves this particular use case, or if some kind of bypass of tokenization still makes sense for mlc-ai/mlc-llm

[jya0]

This doesn't work unfortunately because it first disables the tiktoken and then as a fallback tries to access the transformer library as a fallback to still tokenize the input.

[efriis]

Just kidding - closing! This was done in #19743. You can set check_embedding_ctx_length=False

[jya0]

I love you!!!