Hexaframe

Hexaframe is a tool that implements Go-with-the-Flow with the help of object segmentation from SAMv2, in order to generate videos from video, coordinate path, and text prompts. It is built with Next.js, Shadcn, and Tailwind, with a simple UI and is hosted on Vercel.

The project follows the following steps:

1. The user uploads an image.

2. The user clicks on the object they want to move, which sends a request to a ComfyDeploy server that implements SAMv2 on a ComfyUI workflow that segments the image.

3. The segmented image is shown to the user, which allows him to grab the segmented object and drag it through the desired path. The coordinate path is computed.

4. The user inputs a text prompt.

5. All the inputs are sent to a second ComfyDeploy server that implements Go-with-the-Flow on a ComfyUI workflow that generates the video.

6. The video is shown to the user.

Examples

Simple bear walking through grass

Input image and object path:

Text prompt:

Brown bear walks through tall green grass. He moves its legs with every step, carefully taking a step when its feet touch the ground beneath the grass.

Generated video:

Bear starts behind tree

Input image and object path:

Text prompt:

Brown bear walks through tall grass. He moves its legs with every step, carefully taking a step when its feet touch the ground beneath the grass. If he passes behind a tree, the bear does not suffer any distortion

Generated video:

Getting Started

This is how segmentation should look like:

First, run the development server:

npm run dev
# or
yarn dev
# or
pnpm dev
# or
bun dev

Open http://localhost:3000 with your browser to see the result.

You can start editing the page by modifying app/page.tsx. The page auto-updates as you edit the file.

This project uses next/font to automatically optimize and load Geist, a new font family for Vercel.

Learn More

To learn more about Next.js, take a look at the following resources:

• Next.js Documentation - learn about Next.js features and API.
• Learn Next.js - an interactive Next.js tutorial.

You can check out the Next.js GitHub repository - your feedback and contributions are welcome!

Deploy on Vercel

The easiest way to deploy your Next.js app is to use the Vercel Platform from the creators of Next.js.

Check out our Next.js deployment documentation for more details.