Odd `incus exec` regression with VMs in Debian packaging

[gibmat]

Is there an existing issue for this?

• There is no existing issue for this bug

Is this happening on an up to date version of Incus?

• This is happening on a supported version of Incus

Incus system details

Seen in both Incus 6.0.5 and 6.17.

Reported upstream as bug [#1114918](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1114918).

Instance details

No response

Instance log

No response

Current behavior

No response

Expected behavior

No response

Steps to reproduce

Commit 4e828ca seems to have introduced some sort of regression that breaks running incus exec against a VM. When doing so the message "Error: websocket: bad handshake" is returned. If trying with a container, it works just fine.

Making this more interesting, if the incus-agent binary that's running within the VM is compiled "normally" via

CGO_LDFLAGS_ALLOW="-Wl,-z,now" CGO_ENABLED=0 go build -trimpath -tags=agent,netgo ./cmd/incus-agent/

it works, but when compiled using the dependencies as packaged by Debian it breaks

CGO_LDFLAGS_ALLOW="-Wl,-z,now" CGO_ENABLED=0 GOPATH=/usr/share/gocode/:/home/gibmat/ GOTOOLCHAIN=local GO111MODULE=off go build -trimpath -tags=agent,netgo ./cmd/incus-agent/

The first obvious check was if any libraries that Incus depends on are terribly out of date in Debian, and nothing really jumped out at me. The commit I bisected down to actually removes the use of gorilla/mux in favor of built-in libraries.

I'm not sure what else to try poking at to figure out why this is breaking for Debian's packaging. If it was a "real" regression (for lack of a better word), I'd expect the Zabbly packaging to also be experiencing this issue.

[stgraber]

I've seen a similar report on another distro where the same behavior would happen with a dynamically linked incus-agent.

Can you confirm that in your case the resulting binary is correctly static?
The flags you're passing certainly suggest a static binary should be produced...

[gibmat]

Yes, the incus-agent binaries produced are statically linked. Also, both the host and VM are running up-to-date sid.

[alandoherty]

I'm also getting this error using incus exec with VM's running Fedora 42 Cloud, Ubuntu Noble Desktop and Ubuntu Plucky Desktop. Binaries seem to be dynamically linked. Host is Fedora 42.

Had to move to LXD. Let me know if I can provide more details.

[stgraber]

If you look at our releases, they do include static builds of the agent: https://github.com/lxc/incus/releases/tag/v6.17.0

Replacing the incus-agent on your system with the static one from the release, then restarting your VM should do the trick until you your distro's packager can fix their build.

[PACordonnier]

Hey
I've investigate the issue and I'm quite sure its not related to dynamically or statically linked incus-agent binary.

I've tried incus-agent 6.15 static and dynamic running on a ubuntu 24.04 guest and same symptoms as this issue.
Bug is not present using incus-agent 6.14 nor 6.16. So it has been fixed somehow but couldnt pinpoint where exactly.

For fedora 42 users such as me I think the solution will come when package will be updated or change the incus-agent binary somehow with a more recent one

but you're right @stgraber, fedora packages the rpm and builds the agent dynamically (as of today). Its incorrect but I dont think the issue is related to this

edit: bug is also fixed on 6.17 (static, binary found in this repo releases)

[alandoherty]

Can confirm I was running Incus 6.15, I have switched to LXD for now but can give it another try when 6.16 is available.

[stgraber]

@gibmat do you still have the issue if building the current Incus?
If so, can you share some more reproduction steps so I can try to reproduce the environment here in a Debian VM?