How to debug "could not create new auth interceptor: rpc error: code = DeadlineExceeded"

[seism0saurus]

I have an error with my docker-compose setup.
It worked for some weeks but broke during the last week.

The woodpecker server starts but the agent does not want to connect.
Here is the logs from the agent container.

woodpecker-agent_1   | {"level":"info","time":"2025-06-09T18:23:50Z","message":"shutdown of whole agent"}
woodpecker-agent_1   | {"level":"info","time":"2025-06-09T18:23:50Z","message":"shutdown healthcheck server ..."}
woodpecker-agent_1   | {"level":"error","error":"http: Server closed","time":"2025-06-09T18:23:50Z","message":"cannot listen on address :3000"}
woodpecker-agent_1   | {"level":"info","time":"2025-06-09T18:23:50Z","message":"healthcheck server stopped"}
woodpecker-agent_1   | {"level":"fatal","error":"could not create new auth interceptor: rpc error: code = DeadlineExceeded desc = context deadline exceeded","time":"2025-06-09T18:23:50Z","message":"error running agent"}
woodpecker_woodpecker-agent_1 exited with code 1

I have the following docker-compose.yaml. My loadbalancer is traefik. My woodpecker is connected to the web network, which is connected to traefik container and the woodpacker network for the connection of the agent and the server. I create a runner in the GUI and set the key in the environment variable for the runner in the compose file.

version: '3'

networks:
  web:
    external: true
  woodpecker:

services:
  woodpecker-server:
    image: woodpeckerci/woodpecker-server:v3
    restart: always
    expose:
      - 8000
      - 9000
    networks:
      - web
      - woodpecker
    volumes:
      - ./woodpecker-server-data:/var/lib/woodpecker/
    environment:
      - WOODPECKER_ADMIN=seism0saurus
      - WOODPECKER_FORGEJO=true
      - WOODPECKER_FORGEJO_URL=https://codeberg.org
      - WOODPECKER_HOST=https://woodpecker.example.com
      - WOODPECKER_OPEN=true
      - WOODPECKER_ORGS=myorg
      - WOODPECKER_FORGEJO_CLIENT=myapiclientid
      - WOODPECKER_FORGEJO_SECRET=myapiclientsecret
      - WOODPECKER_LOG_LEVEL=trace
      - WOODPECKER_DEBUG_PRETTY=true
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.woodpecker.rule=Host(`woodpecker.example.com`)"
      - "traefik.http.routers.woodpecker.tls.certresolver=le"
      - "traefik.http.routers.woodpecker.entrypoints=websecure"
      - "traefik.http.routers.woodpecker.service=woodpecker"
      - "traefik.http.services.woodpecker.loadbalancer.server.port=8000"

  woodpecker-agent:
    depends_on:
      woodpecker-server:
        condition: service_healthy
    image: woodpeckerci/woodpecker-agent:v3
    networks:
      - woodpecker
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WOODPECKER_SERVER=woodpecker-server:9000
      - WOODPECKER_AGENT_SECRET=anagenttokenfromthegui
      - WOODPECKER_MAX_WORKFLOWS=1

What exactly does the error message mean? Is it an error between the server and the forgejo instance (codeberg)? Is it an error between the agent and the server?
I could start another image instead of the agent and connect to the server on both ports. So there should be no connection issue.
In another thread with the error someone mentioned I should check the token. Which token? I tried the manual registration and setting the environment variable for both the server and the client.

kind regards
seism0saurus

[qwerty287]

The agent fails to run as it tries to listen at port 3000. Can you try changing the healthcheck address? https://woodpecker-ci.org/docs/administration/configuration/agent#healthcheck_addr

[seism0saurus]

Thank you for your answer. I added WOODPECKER_HEALTHCHECK_ADDR=:7000 as an environment for the agent.
The port changed in the error message but the message is still there. I'm not sure, why the process inside the docker container should be unable to bind to a port inside the container. I also tried some other ports like 8000, 8080, 9000, 7777.
Shouldn't the container has its own portrange so that there are no conflicts with other processes or containers? I seem to be missing something here.

woodpecker-agent_1   | {"level":"info","time":"2025-06-10T12:35:27Z","message":"log level: info"}
woodpecker-agent_1   | {"level":"info","time":"2025-06-10T12:35:27Z","message":"no agent config found at '/etc/woodpecker/agent.conf', start with defaults"}
woodpecker-agent_1   | {"level":"info","time":"2025-06-10T12:35:35Z","message":"shutdown of whole agent"}
woodpecker-agent_1   | {"level":"info","time":"2025-06-10T12:35:35Z","message":"shutdown healthcheck server ..."}
woodpecker-agent_1   | {"level":"error","error":"http: Server closed","time":"2025-06-10T12:35:35Z","message":"cannot listen on address :7000"}
woodpecker-agent_1   | {"level":"fatal","error":"could not create new auth interceptor: rpc error: code = DeadlineExceeded desc = context deadline exceeded","time":"2025-06-10T12:35:35Z","message":"error running agent"}

I also tried to deactivate the healthcheck with WOODPECKER_HEALTHCHECK=false for the agent, but it still crashes, although without the binding error.

woodpecker-agent_1   | {"level":"info","time":"2025-06-10T12:49:13Z","message":"log level: info"}
woodpecker-agent_1   | {"level":"info","time":"2025-06-10T12:49:13Z","message":"no agent config found at '/etc/woodpecker/agent.conf', start with defaults"}
woodpecker-agent_1   | {"level":"info","time":"2025-06-10T12:49:21Z","message":"shutdown of whole agent"}
woodpecker-agent_1   | {"level":"fatal","error":"could not create new auth interceptor: rpc error: code = DeadlineExceeded desc = context deadline exceeded","time":"2025-06-10T12:49:21Z","message":"error running agent"}
woodpecker_woodpecker-agent_1 exited with code 1

[qwerty287]

Ok, this is really weird. Can you try to get into a shell in the container and check which process uses these ports?

EDIT: the error doesn't say that the address is already in use. I can try to check again with the code but I can't do now. I'll try to check tomorrow.

[seism0saurus]

I have news from some experiments:
The same docker-compose.yaml works on a second machine with Debian Trixie. So it's a newer version of docker.
After that I reinstalled docker on the server with sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Before and after the installation the docker.io version was 20.10.24+dfsg1-1+deb12u1+b1. But after the reinstallation the docker-compose.yaml worked again.
I'm not sure, which component of docker on my server was screwed up, because all other things including many different components worked, except the agent.
Thank you very much for your time.