attach image to prompt

[aligit]

in frontend applications we always need to attach images. I couldn't find this information any place.
I'm using the /file but I'm not sure if there is any other solution.

Error 400: {"type":"error","error":{"type":"invalid_request_error","message":"The request body is not valid JSON: str is not valid UTF-8: surrogates not allowed:
line 1 column 1 (char 0)"}}

[olimorris]

This isn't built into the plugin. It's a great suggestion though and something I would like to add in for models that support it.

It's on my roadmap but not sure when I'll get around to it.

Could you share an ideal workflow for how you'd send an image to an LLM with CodeCompanion? Would be good to understand more how people would like it to function.

[ravitemer]

@olimorris MCP servers can give different types of content like text, image, audio, blobs https://spec.modelcontextprotocol.io/specification/2025-03-26/server/tools/#tool-result

E.g - puppeteer server navigates to dev server url and takes screenshots.

mcphub for any tool call returns these different content types as

result = {
  text = "string",
  images = {
         {
                  data = v.data, -- base64 string
                mimeType = v.mimeType or "application/octet-stream",
        }
  },
  audios = {} --similar
  blobs = {} --similar
}

I'm only using result.text as message.content should be a string currently in codecompanion.

                   self.chat:add_buf_message({
                        role = config.constants.USER_ROLE,
                        content = string.format(
                            [[The `%s` call returned the following text: 
%s]],
                            action_name,
                            result.text
                        ),
                    })

Just like how you map codecompanion defined roles (config.constants.roles) to adapter specific roles, we can map custom codecompanion message.content object. Maybe something like

{
 type = "text" 
 text = "What is in this image?"  
}
{
 type = "image" 
 image = {
   type = "base64", -- or url
   base64 = "base64datastring"
}
{
--later build on this for other types if needed
}

This will convert for openai adapter like

 {
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {
                "type": "image_url",
                "image_url": {
                    "url": "base64datastring",
                },
            },
        ],
    }],

[olimorris]

My most common use with Cursor or Chatgpt desktop is using the paste from clipboard feature.

So I usually use screenshoot to clipboard feature from mac

then paste it in chatgpt/cursor.

I personally think it is more efficient than having need to save a screenshoot (removing the need to think where to store the file)

https://github.com/hakonharnes/img-clip.nvim

img-clip have capability for paste image, maybe we can utilize this to load the image into the chat buffer? but I am still unsure how to send the image to the model though

This looks great! I tend to use CleanShot a lot for work and being able to paste directly into apps would be my preferred way of doing things. Img-clip.nvim looks like it could make this possible. Very exciting. I'll make sure I add this into the roadmap after function calling.

[olimorris]

@ravitemer - As always, thanks for your brilliant insight and thoughts. I think for sure this will need to have a structure like you proposed. Getting this to work across multiple adapters will no doubt be "fun" 😉.

[ravitemer]

Getting this to work across multiple adapters will no doubt be "fun" 😉.

🥲 I think, it's not that urgent though. I'm sure you are having enough "fun" with moving-to-function-calling. If you define the codecompanion content type, I am sure me or somebody else will handle updating the adapters. (for now we can leave it to the user to send images to only vision enabled models as validating each model to check if it supports vision might be too much 🤔)

[aligit]

Hey @olimorris ,
Sorry for the late reply. Indeed this is extremely useful for frontend development. I frequently find my self copy the UI from Figma or similar solutions. Claude and chatgpt both support it. I think it should not be tied to the files from the project as Screenshots are not stored per project. I was doing this before moving from avante.nvim to this great plugin. They do it as mentioned by https://github.com/hakonharnes/img-clip.nvim. I'm very happy that is on the roadmap.
Thanks

[pidgeon777]

There is some really interesting info here:

Using vision input in Copilot Chat with Claude and Gemini is now in public preview

Also:

https://docs.github.com/en/copilot/using-github-copilot/copilot-chat/asking-github-copilot-questions-in-your-ide#using-images-in-copilot-chat

[ravitemer]

@pidgeon777

#1105

I have tried to send images with copilot previously. But got the error described in the discussion. I am unable to setup mitmproxy properly in my system in order to try and debug the requests sent by vscode. Can you try if possible?

[hanipcode]

hey guys, I tried to work on this, I tried to build the simplest form of it, which is, just inputing the image url and share it with the AI, and only in openai compatible adapter. it seems I can get it working. I tried to share this image with the AI: https://images.unsplash.com/photo-1744167602422-245a1d448e5e?q=80&w=1887&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D

image2.mp4

I feel the code is a bit hacky though, here is the snippet of the specific part to add image

---@param chat CodeCompanion.Chat
function M.slash_add_image_url(chat)
	local function callback(input)
		if input then
			local id = "<image_url>" .. input .. "</image_url>"
			local new_message = {
				{
					type = "text",
					text = "the user is sharing this image with you. be ready for a query or task regarding this image",
				},
				{
					type = "image_url",
					image_url = {
						url = input,
					},
				},
			}

			local constants = require("codecompanion.config").config.constants
			chat:add_message({
				role = constants.USER_ROLE,
				content = vim.fn.json_encode(new_message),
			}, { reference = id, visible = false })

			chat.references:add({
				id = id,
				source = "adapter.image_url",
			})
		end
	end
	vim.ui.input({ prompt = "> Enter image url", default = "", completion = "dir" }, callback)
end

because content (or maybe internally message (?)) only support string, then I need to do json.encode on the add_message call and re parse to json on the from_message handler.

			form_messages = function(self, messages)
				local result = openai.handlers.form_messages(self, messages)

				local fun = require("hanipcode.local.fun")
				fun.map(result.messages, function(v)
					local ok, json_res = pcall(function()
						return vim.fn.json_decode(v.content)
					end, "not a json")
					if ok then
						v.content = json_res
						return v
					end
					return v
				end)

				return result

it works and doesn't need interface/type change. but I am not sure if this will work in other non openai compatible models. also not sure if there are other unintended implication (I am not fully familiar with the codecompanion internal)

I'll try to tinker it more to make it work with image from clipboard

[hanipcode]

image3.mp4

I finally can paste image to the chat 🚀

here is my adapter code

local M = {}

local default_model = "google/gemini-2.0-flash-001"
local available_models = {
	"google/gemini-2.0-flash-001",
	"google/gemini-2.5-pro-preview-03-25",
	"anthropic/claude-3.7-sonnet",
	"anthropic/claude-3.5-sonnet",
	"openai/gpt-4o-mini",
}
local current_model = default_model

function M.select_model()
	vim.ui.select(available_models, {
		prompt = "Select  Model:",
	}, function(choice)
		if choice then
			current_model = choice
			vim.notify("Selected model: " .. current_model)
		end
	end)
end

---@param chat CodeCompanion.Chat
---@param id string
---@param input string
function M.add_image(chat, id, input)
	local new_message = {
		{
			type = "text",
			text = "the user is sharing this image with you. be ready for a query or task regarding this image",
		},
		{
			type = "image_url",
			image_url = {
				url = input,
			},
		},
	}

	local constants = require("codecompanion.config").config.constants
	chat:add_message({
		role = constants.USER_ROLE,
		content = vim.fn.json_encode(new_message),
	}, { reference = id, visible = false })

	chat.references:add({
		id = id,
		source = "adapter.image_url",
	})
end

---@Param chat CodeCompanion.Chat
function M.slash_paste_image(chat)
	local clipboard = require("img-clip.clipboard")
	local paste = require("img-clip.paste")
	local prefix = paste.get_base64_prefix()
	local base64res = clipboard.get_base64_encoded_image()
	local url = prefix .. base64res
	local hash = vim.fn.sha256(url)
	local id = "<pasted_image>" .. hash:sub(1, 16) .. "</pasted_image>"
	M.add_image(chat, id, url)
end

---@param chat CodeCompanion.Chat
function M.slash_add_image_url(chat)
	local function callback(input)
		if input then
			local id = "<image_url>" .. input .. "</image_url>"
			M.add_image(chat, id, input)
		end
	end
	vim.ui.input({ prompt = "> Enter image url", default = "", completion = "dir" }, callback)
end

function M.get_slash_commands()
	return {
		["image_url"] = {
			callback = M.slash_add_image_url,
			description = "add image via url",
		},
		["image_paste"] = {
			callback = M.slash_paste_image,
			description = "add image from clipboard",
		},
	}
end

function M.get_adapter()
	local openai = require("codecompanion.adapters.openai")
	return require("codecompanion.adapters").extend("openai_compatible", {
		env = {
			url = "https://openrouter.ai/api",
			api_key = "OPENROUTER_API_KEY",
			chat_url = "/v1/chat/completions",
		},
		handlers = {
			form_parameters = function(self, params, messages)
				local result = openai.handlers.form_parameters(self, params, messages)
				return result
			end,
			form_messages = function(self, messages)
				local result = openai.handlers.form_messages(self, messages)

				local fun = require("hanipcode.local.fun")
				fun.map(result.messages, function(v)
					local ok, json_res = pcall(function()
						return vim.fn.json_decode(v.content)
					end, "not a json")
					if ok then
						v.content = json_res
						return v
					end
					return v
				end)

				return result
			end,
		},
		schema = {
			model = {
				default = current_model,
			},
		},
	})
end

return M

[hanipcode]

added keymap to paste the image so no need to write /image_paste

vim.keymap.set("n", "<leader>i", function()
	local adapter = require("hanipcode.local.adapter")
	local codecompanion = require("codecompanion")
	local chat = codecompanion:last_chat()
	if not chat then
		vim.notify("Chat not exist")
	end
	adapter.slash_paste_image(chat)
end)

[pidgeon777]

Would it be possible to also adapt it for the GitHub Copilot provider? Please see here:

#1208 (comment)

[daviareias]

What about creating images? How are you guys working with that? I guess neovim doesn't display the created image response

[pidgeon777]

I'm also curious.

[olimorris]

I tend to need this infrequently, mostly for trivial stuff, so I just use ChatGPT or Midjourney. The latter has been great for generating backgrounds that I use in videos as it's editing tools are pretty decent.

[pidgeon777]

I think Neovim somehow already supports displaying images in buffer, maybe through the Kitty Graphics Protocol? Also, Neovide is trying to add such feature, but I don't know if it will be possible to do so also for Windows.

https://sw.kovidgoyal.net/kitty/graphics-protocol/

neovim/neovim#30889

neovide/neovide#2088

[ravitemer]

If you have a terminal that supports kitty graphics protocol, you can setit up with image.nvim plugin to view images directly in the codecompanion buffer. But most of the time, it's irritating to look at and navigate in the buffer. You can gx on that image path and open it in the system preview.

By default gx in codecompanion buffer is used to clear chat, so maybe you can map it to something else like gl to clear and gx to open links:

require("codecompanion").setup({
	strategies = {
		chat = {
			keymaps = {
				clear = {
					modes = {
						n = "gl",
					},
				},
			},
		},
	},
})

Here is a demo of using Image-gen MCP server to generate a gx to preview the image:

cc-image-gen.mp4