Port embedding API to plugin

[AleksKnezevic]

We need to port embedding API support in the vLLM plugin that will be part of this repo #1365.

[mmanzoorTT]

• Embedding API is implemented and working end-to-end for Qwen3-4B model.
• All Out Of Memory errors are fixed.
• I can run large sequences up till 8K tokens. It seems that this limitation is due to vLLM model runner implementation. investigating now.

[mmanzoorTT]

vLLM requires additional options/flags to process larger token inputs. However, device ran out of memory for larger tokens. We investigated and found following solutions/optimizations to run input tokens of length 32K.

• Adding support for scalar version of TTNN::PowOp which uses scalar as exponent instead of a tensor. This freed up ~1.3GB of space and crucial part to run larger model. PR is currently under review.
• Removing unnecessary typecast ops. upstream PR and Decomposition pass fixes. This helps to avoid space utilization due to unnecessary upcasts and improves the performance as well.
• ConstEval optimization: This optimization is only tested by manually changing/rewriting the graph. This gives us additional ~500MB space. Qwen3-Embedding-4B model can run tokens of length 32K without this optimization; however, it may be beneficial for some other model.

[mmanzoorTT]

vLLM Embedding API update

• Pre-compilation of multiple binaries for different input sequence lengths.
• Process multiple batches; A batch contains one or more input sequences.
• Next step is to include attention_mask; currently, we are relying on tt-metal default attention_mask

[bgoelTT]

@mmanzoorTT question about the pre-compilation

From the perf results I've seen it looks like we are compiling programs for each of the Power-of-Two (PoT) input sequence lengths. In the tt-metal implementation of LLMs / transformers we pad input requests to these PoT sizes so that we can reuse these pre-compiled binaries.

Will the Pooling Model Runner here also perform such padding?

[AleksKnezevic]

@LPanosTT added that part, so he can better comment.

[LPanosTT]

@bgoelTT Yes the precompile step will do this for the model backbone.

[bgoelTT]

Okay understood, and where does the request padding take place?