Allow video upload to models like gemini recreate animations

[rrfaria]

Ability to upload video files directly into the Roo Code interface, allowing multimodal models (like Gemini 2.5 Pro) to analyze keyframes and recreate animations via code (CSS/JS/SVG).

Animation examples:

design.mp4

[jordanrendric]

Was gonna open an issue for this, but then I saw yours was already open.

So, quick update: I actually found a workaround for it. I just made a .aiAuxFiles folder in the project, threw it in the .gitignore (so my commits don't get bloated with a ton of vids), and then I just drop the vids I wanna send to Gemini in there. Then I just mention the file in chat, and it can see the video. Not perfect, but hey, it works.

This feature'd also be super useful for FE debugging, especially for those bugs that are harder to explain than they are to believe even exist.

[VooDisss]

@jordanrendric I'm using gemini 2.5 pro. I'm working Roo source code today, so it would handle attached video files... Not much luck yet, sunk about 4 hours into it already...

[jordanrendric]

Just forked the repo to work on it locally a little over an hour ago, and nothing better than using Roo to hack on Roo itself lol.

Turns out Gemini has a specific API for files (including videos) larger than 20MB, and smaller ones can be sent inline as base64 (same as how image uploads work). I’ve been running some tests, got a partial success, but it looks like only the first frame went through. Now I’m making Roo code Roo to send videos the way Gemini’s docs recommend:
https://ai.google.dev/gemini-api/docs/video-understanding

Also noticed Roo uses the Anthropic-style message format and converts it to Gemini’s format, pretty clever.

[jordanrendric]

Well well well… turns out it wasn’t that hard after all 😏
Got a working output from my test (even with videos over 20MB!)

Here’s the (translated) output Gemini gave me, the original was in Portuguese.
(Yeah, I know, being Brazilian means I already spend 15 hours/day thinking in English… so at least when I’m on my IDE or chatting with my AI agents, I stick to Portuguese to not forget how to speak my native language 😂)

Translation:

In addition to the text with patient info and the prescription, the video shows the UI of an app — apparently an electronic health record system or some kind of medical platform.
After looking at the interface, I spotted the following:
Theme & Design: dark mode with green highlights (icons, buttons) — modern look.
Main Navigation: a bottom nav bar with 4 sections:
Home (house icon): probably dashboard
Patients (people icon): patient list/management
Support (gear icon): settings/help
Menu (hamburger icon): extra options
Interaction Screen: seems to be a chat/assistant interface
Active Context: a tag saying "Context: Jordan Vasconcelos", meaning the system is currently focused on this specific patient
[...]

This was actually a screen recording I made to report a bug in the chat interface of an app I’m working on, and yeah, Gemini did manage to understand it and described the video pretty well. 🤯

[VooDisss]

Yes! You did it very fast... I could not make it work... Can you push the changes to your repository, please? I wonder what changed there...
Need to push it to the Roo Code main branch (after de-vibing it :D )

[jordanrendric]

Absolutely! Gonna clean up the vibe-inspired hacks first 😅 and push everything soon.

Also, here’s a breakdown of what Roo actually did to Roo, figured it’d be helpful to detail what changed across the codebase to support video uploads.

Roo explaining what it did to the Roo code… well, to Roo itself 😂

Goal: allow video uploads (in addition to images), but only when a Gemini-family model (that supports video understanding) is selected.

---

1. webview-ui/src/components/common/Thumbnails.tsx → MediaThumbnails.tsx

This was the core frontend change.

• Renamed: from Thumbnails to MediaThumbnails to reflect it now handles multiple media types. Internal props like images were renamed to mediaItems.
• MIME detection: added a getMimeType util to extract the type (like image/png or video/mp4) from the data: URI.
• Conditional rendering:
  • If it's a video (video/...), render a <video> tag with muted, loop, and autoPlay for animated preview.
  • If it's an image (image/...), continue using <img>.

---

2. webview-ui/src/components/chat/ChatView.tsx

This component controls the chat logic.

• Dynamic file types: added logic (via useMemo) to check the current AI model (apiConfiguration.apiModelId).
  • If model includes "gemini-2.5-pro" or "gemini-1.5-flash", allow video file types (mp4, mov, etc.).
  • Otherwise, only allow images.
• State rename: selectedImages → selectedMedia.
• Prop passing: acceptedFileTypes and selectedMedia now passed down to ChatTextArea.

---

3. webview-ui/src/components/chat/ChatTextArea.tsx

• New props: receives acceptedFileTypes, selectedMedia, and setSelectedMedia.
• Dynamic validation: in handlePaste and handleDrop, replaced static image check with dynamic list. Now accepts type === "image" or type === "video".
• New component: replaced <Thumbnails /> with <MediaThumbnails />, using updated props.

---

4. TaskHeader.tsx and ChatRow.tsx

Both showed media previews.

• Quick update: switched to import and use the new <MediaThumbnails /> to ensure consistent rendering.

---

5. webview-ui/src/components/chat/__tests__/ChatTextArea.spec.tsx

Unit test updates.

• Prop fixes: since selectedMedia, setSelectedMedia, and acceptedFileTypes became required props, updated defaultProps in tests accordingly to avoid breakage.

---

In short: made file acceptance logic dynamic, built a more flexible media preview component, and updated all related components/tests to support the new flow.
Pretty happy with how clean it turned out, pushing soon and will ping here once it’s live! 😄

[jordanrendric]

@VooDisss just pushed everything to my fork!

You can check it out here: https://github.com/jordanrendric/Roo-Code (branch: main)

[VooDisss]

@jordanrendric I found one issue in your solution - it does not take into account the user prompt, when sending it together with a video and I could not find the solution (again, I'm not lucky and I have tried for around 1 hour).
This video support is extremely useful for debugging/describing UX or UI problems and I have made "UX Video Analyzer" mode for it, but I have a problem of it not receiving user prompt text. See the answer it gave:

[jordanrendric]

@VooDisss Was this test using the mode you created?
If so, maybe the system prompt is too restrictive, would be useful to share the SP so we can take a look and see what could be improved to make the model follow both the SP and the user prompt more reliably.

And about the PR upstream: let’s do it.
I’ll add you on Discord!
I had a few personal things and other work yesterday, so I couldn’t give GH the proper attention.

[VooDisss]

@jordanrendric Yes, the test was using the mode I created using "Mode writer" from roocode, I made about 8 iterations of it until I feel confident it should perform very well if the prompt would reach together with the video (multimodal API request it has to send). I believe system prompt is not related to this issue...

I will be able to send the .XML rules (that the mode uses) and .roomodes file once we get in contact.

I have spent about another 6 hours trying to solve this issue and no luck - any model I use (2.5 pro/flash and kimmi-k2:free)gets into loops and gets no result.

[jordanrendric]

One thing about my code mod: it recognizes the gemini 2.5 models, and then enables the sending of videos, so it may be that with a model like the kimmi k2 it does not actually send the video to the model

[jordanrendric]

@VooDisss Just added you on Discord.
I’m in the middle of launching a web app right now, and turns out some “competitors” (rats) are dropping a ripoff of what I’m building… and worse: they did it in WordPress. Absolute rats, pigs, and newbies. 🤮🤮🤮

So yeah, today’s probably gonna be hectic on my end, but tonight I’ll be 100% available to sit down and code our solution. Shouldn’t take more than 1–2 hours to wrap it up.