"No targets could be found for kubernetes service"after upgrade to Kong 3.6 and KIC 3.1

[mloskot]

Hi,

I have upgraded Kong on my AKS cluster:

• Kong from 3.3 to 3.6
• KIC from 2.9 to 3.

and the ingress-controller logs are full of

2024-03-06T09:44:34Z	info	No targets could be found for kubernetes service	{"v": 0, "namespace": "test", "name": "geo", "kong_service": "test.geo.4331"}
2024-03-06T09:44:34Z	info	No targets found to create upstream	{"v": 0, "service_name": "test.geo.4331"}

However, my services/apps are accessible, that is, Kong routes the traffic just fine.

I have not found any issues or discussions that could explain why I'm receiving those info-s.
Could anyone help me understand what is going on?

[chlislb]

any workaround about this issue? I met the same issue.

[mloskot]

If more people experience this, it may be a good idea to "Create issue from discussion" ?

[pmalek]

@mloskot Are you able to provide minimal reproduction steps for this one? This would happen when the underlying k8s Service has not EndpointSlices which are marked as Ready.

[chlislb]

@pmalek , from my case, when the lead of ingress controller was evicted or restart, the new lead will collect all services, endpointslices and others things, then it will report the issue. I think it should be caused by controllers.Service workers collect service slower.
logs:
2024-03-15T06:54:32Z error Failed to fetch service {"service_name": "", "service_namespace": "kong", "error": "Service kong/kong-log2 not found"}
2024-03-15T06:54:32Z debug Service has zero k8sServices backends, cannot generate tags for it properly {"v": 1, "service": "kong.kong-log2.31029"}
2024-03-15T06:54:32Z error can't add target for backend kong-log2: no kubernetes service found {"name": "kong-log2-data", "namespace": "kong", "GVK": "networking.k8s.io/v1, Kind=Ingress", "error": "resource processing failed"}

[mloskot]

@pmalek I'm not sure what to provide really. I did an upgrade and that's it.

This would happen when the underlying k8s Service has not EndpointSlices which are marked as Ready.

I don't think I have any EndpointSlices configured, not explicitly. The only one I see configured is

$ kubectl get endpointslice
NAME         ADDRESSTYPE   PORTS   ENDPOINTS      AGE
kubernetes   IPv4          443     51.131...18    178d

[pmalek]

@mloskot @chlislb A list of reproducible instructions would be ideal. If you've installed with a helm chart then the incantation of the install command + values.yaml and the minimal set of manifests that you add on top of that which can trigger the described behavior.

[mloskot]

@pmalek thank you for the will to help out!

and the minimal set of manifests that you add on top of that which can trigger the described behavior.

This will be tricky, but I will try to prepare some ASAP, it may take a few days as I'm a bit swamped at the moment.

If you've installed with a helm chart then the incantation of the install command + values.yaml

I deploy Kong with Flux, here is the whole manifest

---
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: kong
  namespace: kong
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled
spec:
  releaseName: kong
  interval: 5m
  chart:
    spec:
      chart: kong
      version: "2.38.0"
      sourceRef:
        kind: HelmRepository
        name: kong
        namespace: flux-system
      interval: 5m
  values:
    deploymentAnnotations:
      kustomize.toolkit.fluxcd.io/prune: disabled
    podAnnotations:
      kustomize.toolkit.fluxcd.io/prune: disabled
    nodeSelector:
      kubernetes.io/os: linux
    deployment:
      kong:
        enabled: true
      serviceAccount:
        create: true
        annotations:
          kustomize.toolkit.fluxcd.io/prune: disabled
    image:
      repository: kong
      tag: "3.6"
    proxy:
      enabled: true
      type: LoadBalancer
      externalTrafficPolicy: Local
      http:
        enabled: true
        servicePort: 80
        containerPort: 8000
        parameters: []
      tls:
        enabled: true
        servicePort: 443
        containerPort: 8443
        parameters: []
      annotations:
        kustomize.toolkit.fluxcd.io/prune: disabled
    ingressController:
      enabled: true
      image:
        repository: kong/kubernetes-ingress-controller
        tag: "3.1"
      env:
        log_level: info
      annotations:
        kustomize.toolkit.fluxcd.io/prune: disabled
    manager:
      enabled: true
      http:
        enabled: true
        servicePort: 8002
        containerPort: 8002
        parameters: []
      tls:
        enabled: true
        servicePort: 8445
        containerPort: 8445
        parameters: []
      annotations:
        kustomize.toolkit.fluxcd.io/prune: disabled
    portal:
      enabled: true
      http:
        enabled: true
        servicePort: 8003
        containerPort: 8003
        parameters: []
      tls:
        enabled: true
        servicePort: 8446
        containerPort: 8446
        parameters: []
      annotations:
        kustomize.toolkit.fluxcd.io/prune: disabled
    portalapi:
      enabled: true
      http:
        enabled: true
        servicePort: 8004
        containerPort: 8004
        parameters: []
      tls:
        enabled: true
        servicePort: 8447
        containerPort: 8447
        parameters: []
      annotations:
        kustomize.toolkit.fluxcd.io/prune: disabled
    env:
      # https://github.com/Kong/kong/blob/master/kong.conf.default
      database: "off"
      nginx_worker_processes: "2"
      nginx_proxy_proxy_buffer_size: 128k
      nginx_proxy_proxy_buffers: 64 128k
      nginx_proxy_large_client_header_buffers: 64 128k
      proxy_access_log: /dev/stdout
      admin_access_log: /dev/stdout
      admin_gui_access_log: /dev/stdout
      portal_api_access_log: /dev/stdout
      proxy_error_log: /dev/stderr
      admin_error_log: /dev/stderr
      admin_gui_error_log: /dev/stderr
      portal_api_error_log: /dev/stderr
      prefix: /kong_prefix/
      log_level: notice

[chlislb]

I think this issue related to #2315.

[chlislb]

more information, in our k8s cluster, the nginx controller with lots of proxied services, but kong ingress controller only have several proxied services.