DriverNetwork IP is not used to in NOMAD_IP variables

[vsarunas]

Nomad version

Nomad v1.11.0-dev

Operating system and Environment details

Issue

Hi @tgross,

Hopefully you're able to help; I am trying to create a basic task driver for running containers on macOS natively using https://github.com/apple/container. The difference here is that it gives a dedicated VM and IP address for each container started.

Driver capabilities which are set:

	capabilities = &drivers.Capabilities{
		SendSignals:         true,
		Exec:                false,
		FSIsolation:         fsisolation.Image,
		NetIsolationModes:   []drivers.NetIsolationMode{drivers.NetIsolationModeNone},
		MustInitiateNetwork: false,
	}

DriverNetwork returned from the plugin:

	network := &drivers.DriverNetwork{
		IP:            h.containerIP,
		AutoAdvertise: true,
	}

The end result is that the running workload does not get advertised the real IP address returned by the driver:

Allocation Addresses:
Label  Dynamic  Address
*http  yes      192.168.2.2:24324

The service is advertised in Consul with the correct IP; but there is one service that requires to know IP on startup that I have trouble bringing up (Kafka broker advertise).

Is there any way to pass the IP returned by the driver to environment of the task if not using dynamic addresses?

Reproduction steps

The same can be reproduced with the mock driver.

Expected Result

The job should show that the IP allocated to the container is 1.1.1.1:

Using IP of 1.1.1.1

Actual Result

The IP of the client which executed the job is shown instead:

Using IP of 192.168.2.2

Job file

job "mock-ip" {
  type = "service"
  group "mock" {
    network {
      #mode = "bridge"
      port "http" {}
    }
    task "mocktask" {
      driver = "mock_driver"
      config {
        run_for          = "1h"
        exit_code        = 0
        driver_advertise = "true"
        driver_ip        = "1.1.1.1"
        stdout_string    = "Using IP of ${NOMAD_IP_http}"
      }
      service {
        name         = "mock"
        port         = "http"
        address_mode = "driver"
      }
    }
  }
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[jrasell]

Hi @vsarunas and thanks for raising this issue. The engineering team triage issues on rotation, so there shouldn't be a need to ping individual engineers.

Without looking and reproducing with the task driver it's hard to fully identify what is happening. Something to look at would be the task env vars which are injected into the task at runtime. These contain a number of network related details which might include the information you're looking for.

Looking through the code:

• the driver network object will get persisted to the env builder here:

  nomad/client/allocrunner/taskrunner/task_runner_getters.go

  Line 113 in e1c576d

  tr.envBuilder.SetDriverNetwork(handle.net)

• the env network related variables get populated here:

  nomad/client/taskenv/env.go

  Line 992 in e1c576d

  func buildNetworkEnv(envMap map[string]string, nets structs.Networks, driverNet *drivers.DriverNetwork) {

[tgross]

@vsarunas there's two different problems here I think:

• Whether the NOMAD_IP variables are being set. I think from your description of the goal that this is the more important piece.
• Whether the driver network IP is being exposed in the alloc status

I don't think the first is solvable except inside the driver. Nomad has to pass an environment in the *TaskConfig it hands to StartTask. So Nomad won't get the drivers.DriverNetwork until after the driver has started the task! If the driver needs to modify the IP, it'll need to override those environment variables itself. After a quick review of the code in task_runner_getters.go and its callers, I think that's only used for writing to state afterwards (which helps during the restore process when a client restarts). It can't set the environment variables of the task that's already started.

The second is trickier, because that data is coming from the allocated resources assigned by the scheduler , which is too early for the task driver's logic to kick in. That will generally speaking be the IP address of the host or the host_network.