Support SmolVLM-256M/500M-Instruct

[isbee]

I was able to run mlx-community/SmolVLM-Instruct-4bit on mlx-swift-examples (VLMEval) by simply updating the modelConfiguration, and it generated responses without any issues.

However, when trying to run mlx-community/SmolVLM-256M-Instruct-{} or mlx-community/SmolVLM-500M-Instruct-{}, the inference crashes with the following MLX error:

MLX error: Shapes (1,576,768) and (1,1024,768) cannot be broadcast. at /Users/user/Library/Developer/Xcode/DerivedData/mlx-swift-examples-djkwqspwvzjyvodqeyxeahqxpsse/SourcePackages/checkouts/mlx-swift/Source/Cmlx/include/mlx/c/ops.cpp:31

Since these models are still based on Idefics3, I expected them to work similarly. This looks like a shape mismatch issue, which might be related to mlx-vlm or mlx-swift.

[yukiarimo]

+1

[Blaizzy]

Hey guys,

I know what happened!

It's related #191

[Blaizzy]

@isbee could you give me the steps to replicate it?

From the build till execution.

I fixed the issue but I'm not a expert swift dev so I need to run it.

[Blaizzy]

Just to validate the code.

[isbee]

I'm using latest mlx-swift-examples commit.

I added below code after this line.

    static public let smolvlminstruct256Mbf16 = ModelConfiguration(
        id: "mlx-community/SmolVLM-256M-Instruct-bf16",
        defaultPrompt: "Describe the image in English"
    )

I change modelConfiguration to smolvlminstruct256Mbf16, and then I run application, got crash.

By the way, it seems like mlx-swift-examples not depend on mlx-vlm and just ported mlx-vlm's idefics3.py to idefics3.swift. So it might be mlx-swift-examples issue? 😅

[Blaizzy]

Yes, I sent a PR to fix idefics3 processing and it should probably also fix this issue.

Here:

ml-explore/mlx-swift-examples#192

[Blaizzy]

Closing this issue and moving it there.