How to use Transformers library in client.baml? to load any safetensor/ggml model and use them.

[vinaayakha]

Novice Enthusiast here,

Asking dumb questions.

I'm using layoutLM and Pillow but I want to use baml library to extract structured data from any document that the user might give. How can I load the Transformers library in baml.client.

There is an option to use vLLM but vLLM is for LLM text generation.
I want to create a baml workflow using layoutLM to extract structured data using layoutLM and continue with summarization using llama or qwen.

Thank you

[imalsogreg]

That's a great question! BAML doesn't support using transformers directly, but I believe you can export your model fromtransformers into GGUF format and then serve it with ollama. BAML knows how to communicate with ollama for language tasks and how to pass in multi-modal data.

I started reading about layoutLM and noticed that it expects bounding boxes as input in a specific format. Do you know if those are required for your use case? As far as I understand, this doesn't exactly match BAML's usual interface to the model provider, but we might be able to figure out a solution.