Users may need to ignore `127.0.0.1` in SSH config to use Podman 5.3

[calebevans]

Issue Description

After updating to 5.3, I started receiving the following error:

caevans@caevans-mac reportportal-connector % podman --log-level=debug ps                            
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps) 
DEBU[0000] ssh_config alias found: 127.0.0.1            
DEBU[0000]   User: core                                 
DEBU[0000]   Hostname: ssh://core@127.0.0.1:49744/run/user/501/podman/podman.sock 
DEBU[0000]   Port: 49744                                
DEBU[0000]   IdentityFile: "~/.ssh/id_ed25519"          
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory
DEBU[0000] Shutting down engines

I found that adding !127.0.0.1 and !localhost to the wildcard rule resolved my issue:

Host * !127.0.0.1 !localhost
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519

Steps to reproduce the issue

Steps to reproduce the issue

1. Install Podman 5.3
2. Don't exclude 127.0.0.1 or localhost in your SSH config rules
3. Execute a Podman command

Describe the results you received

OUTPUT

caevans@caevans-mac reportportal-connector % podman --log-level=debug ps                            
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called ps.PersistentPreRunE(podman --log-level=debug ps) 
DEBU[0000] ssh_config alias found: 127.0.0.1            
DEBU[0000]   User: core                                 
DEBU[0000]   Hostname: ssh://core@127.0.0.1:49744/run/user/501/podman/podman.sock 
DEBU[0000]   Port: 49744                                
DEBU[0000]   IdentityFile: "~/.ssh/id_ed25519"          
Cannot connect to Podman. Please verify your connection to the Linux system using `podman system connection list`, or try `podman machine init` and `podman machine start` to manage a new Linux VM
Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory
DEBU[0000] Shutting down engines

Describe the results you expected

Expected to execute Podman commands without modifying ssh config.

podman info output

host:
  arch: arm64
  buildahVersion: 1.38.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.12-2.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.12, commit: '
  cpuUtilization:
    idlePercent: 99.76
    systemPercent: 0.12
    userPercent: 0.12
  cpus: 4
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2044
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.11.3-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 7222734848
  memTotal: 7716302848
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.2-2.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.2
    package: netavark-1.12.2-1.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.2
  ociRuntime:
    name: crun
    package: crun-1.17-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.17
      commit: 000fa0d4eeed8938301f3bcf8206405315bc1017
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240906.g6b38f07-1.fc40.aarch64
    version: |
      pasta 0^20240906.g6b38f07-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: unix:///run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 8h 28m 41.00s (Approximately 0.33 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 7329873920
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 49
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.3.0
  Built: 1731456000
  BuiltTime: Tue Nov 12 17:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.7
  Os: linux
  OsArch: linux/arm64
  Version: 5.3.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

SYSTEM INFO

• macOS 15.1
• Apple Silicon CPU
• Podman:

caevans@caevans-mac ~ % podman version
Client:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.23.3
Git Commit:   874bf2c301ecf0ba645f1bb45f81966cc755b7da
Built:        Tue Nov 12 09:10:17 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.3.0
API Version:  5.3.0
Go Version:   go1.22.7
Built:        Tue Nov 12 17:00:00 2024
OS/Arch:      linux/arm64

Additional information

I found that adding !127.0.0.1 and !localhost to the wildcard rule resolved my issue:

Host * !127.0.0.1 !localhost
  UseKeychain yes
  AddKeysToAgent yes
  IdentityFile ~/.ssh/id_ed25519

[Luap99]

Fix in #24568, we are planning for a 5.3.1 release early next week with some other fixes

[afbjorklund]

Was the root cause here that we didn't tilde-expand the key path (for Host *)?

Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_ed25519": open ~/.ssh/id_ed25519: no such file or directory

The workaround should be fine to fix the regression, but it should cope with:

Host *
    IdentityFile ~/.ssh/id_rsa

But it doesn't:

Error: unable to connect to Podman socket: failed to read identity "~/.ssh/id_rsa": open ~/.ssh/id_rsa: no such file or directory

Even if it is the default (and thus was never tested, when it came to the parser)

• Add support for ssh_config for connection #23847

---

Like so:

--- a/pkg/bindings/connection.go
+++ b/pkg/bindings/connection.go
@@ -189,8 +189,13 @@ func sshClient(_url *url.URL, uri string, identity string, machine bool) (Connec
                }
        }
        if val := cfg.Get(alias, "IdentityFile"); val != "" {
+               h, err := os.UserHomeDir()
+               if err != nil {
+                       return connection, fmt.Errorf("user homedir could not be determined: %w", err)
+               }
                if val != ssh_config.Default("IdentityFile") {
                        identity = strings.Trim(val, "\"")
+                       identity = strings.Replace(identity, "~", h, 1)
                        found = true
                }
        }

Then it goes back to failing "properly", i.e. due to overriding the ssh key for all hosts.

Error: unable to connect to Podman socket: failed to connect: ssh: handshake failed: ssh: unable to authenticate, attempted methods [none publickey], no supported methods remain

[afbjorklund]

The real issue was that we only had a single key, and didn't support multiple.

It was supposed to start with the Host *, and then continue with the Host...

• Add support for retrieving all IdentityFile directives kevinburke/ssh_config#28

That is why the podman connection failed, even though ssh is still successful:

debug1: Next authentication method: publickey
debug1: Offering public key: /home/anders/.ssh/id_rsa RSA SHA256:... explicit agent
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Offering public key: /home/anders/.lima/_config/user ED25519 SHA256:... explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60

Changing the order of the ssh config lines (so that Host * comes after) makes it work.

The current ssh_config parser returns the first key that it can find, but only a single one.

---

It still needs to expand tilde on the path, though:

• Add support for ~ references to home directories in Include kevinburke/ssh_config#31

And apparently there are bugs in support for the !?

The string * !127.0.0.1 !localhost matches all hosts...

[Luap99]

@afbjorklund There are multiple issues, see the commits in the PR. But the biggest issue from your PR was that you overwrote settings that were already set in the ssh URL. For the machine connection we already have the proper user, port and identity file set so it should not pick the ones from the shh config as these are defaults not overwrites.