Assign public IPs to workers in multi-cloud cluster

[jakoberpf]

Hey peoples,

I am stuck on expanding a multi-cloud deployment with some worker nodes in the Oracle Cloud. My controlplane is running in Hetzner and I have some Proxmox VMs, Strato VServers and AWS EC2s as workers, all work great. Now I want to add some Oracle VMs. If I run talosctl get --insecure nodeaddresses --nodes 123.123.123.123 (123.123.123.123 is a Oracle VM booted up with TalosOS) I get

NODE   NAMESPACE   TYPE          ID             VERSION   ADDRESSES
       network     NodeAddress   accumulative   2         ["10.0.12.84/24"]
       network     NodeAddress   current        2         ["10.0.12.84/24"]
       network     NodeAddress   default        1         ["10.0.12.84/24"]
       network     NodeAddress   routed         2         ["10.0.12.84/24"]

because the networking on Oracle, the VM does not know it public IP. I can assign the public IP to the certSANs

  certSANs:
    - 10.0.12.84
    - 123.123.123.123

to make the certificate working, but once I apply the machine config talosctl apply-config --nodes 123.123.123.123 --file worker-123.123.123.123.yaml --insecure the API is not reachable anymore because Talos is not configured to listen on the public IP 123.123.123.123 and it does not know about it from the nodeaddresses.

talosctl edit machineconfig --nodes 123.123.123.123
# rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 123.123.123.123:50000: i/o timeout"

How can I provider this public IP in the machine configuration? I tried different variations of (only the custom part of the config)

version: v1alpha1
debug: true
persist: true
machine:
  type: worker
  certSANs:
    - 10.0.12.84
    - 123.123.123.123
  kubelet:
    nodeIP:
      validSubnets:
        - 123.123.123.123/32
  network:
    interfaces:
      - interface: enp0s6
        addresses:
          - 10.0.12.84/24
          - 123.123.123.123/32
        dhcp: true
    kubespan:
      enabled: true

[smira]

because the networking on Oracle, the VM does not know it public IP.

If using oracle image of Talos, it should automatically pull its public IP.

rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 123.123.123.123:50000: i/o timeout"

This depends on Oracle Cloud, not Talos. Usually in the clouds public IPs are not assigned to the machine, but rather mapped 1:1 on the infrastructure side.

The way you assigned public IP is correct for Talos Linux, but not sure how it should work with Oracle Linux.

[jakoberpf]

@smira Thanks that actually makes sence to me. I am using the oracle type of Talos image of version 1.8.4. Is there a troubleshooting step I can do to get close why I don't see the public IP in talosctl get --insecure nodeaddresses? From you elaboration if should be present there, since Talos would pull the IP from the metadata for eg.

And there is no forceable way to assign a IP if Talos is not able to detect it?

[smira]

Looking at the code (

talos/internal/app/machined/pkg/runtime/v1alpha1/platform/oracle/oracle.go

Lines 1 to 7 in 8252dca

	// This Source Code Form is subject to the terms of the Mozilla Public
	// License, v. 2.0. If a copy of the MPL was not distributed with this
	// file, You can obtain one at http://mozilla.org/MPL/2.0/.
	// Package oracle provides the Oracle platform implementation.
	package oracle

), I don't see it pulling a public IP like it does e.g. on AWS.

Once again, I'm not that much familiar with Oracle Cloud to answer this. This is a community supported platform, so it might need some changes if Oracle has support now for some new features (or not).

The docs we have use a load balancer to expose private IPs to the world.

[jakoberpf]

Yeah I tested that with https://docs.oracle.com/de-de/iaas/Content/Compute/Tasks/gettingmetadata.htm and the Oracle Metadata unfortunantly does not include the public IPs of an interface. From you elabotration I understand that currentl the approach of Talos OS is the get all the information from the metadata, but would it be technically possible to implement a override functionality into the machine config? I am not in the development world of Talos (yet) so I could not judge.

[smira]

Please see my response above - I don't think you want that IP on the machine itself, but rather it's mapped, and you need to update cloud firewall to update connections.

If you get network connection to the Talos machine, you should get an error about IP mismatch in the certificate, then you can easily patch it in via .machine.certSANs.