Error Vespa Config Server

[monchopena]

Hi 👋,

This is the log:

[2024-06-04 20:13:04.033] INFO    configserver     Container.com.yahoo.jdisc.http.server.jetty.JettyHttpServer  Threadpool size: min=32, max=32
[2024-06-04 20:13:04.926] INFO    config-sentinel  sentinel.config.frt.frtconfigagent   No response / error from config server. This is normal before an application package is deployed. (key: name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local) (errcode=103, validresponse:0), trying again in 6.000000 seconds
[2024-06-04 20:13:05.124] INFO    configserver     Container.com.yahoo.container.jdisc.state.StateMonitor       Changing health status code from 'initializing' to 'up'
[2024-06-04 20:13:05.246] INFO    configserver     Container.com.yahoo.container.handler.threadpool.ContainerThreadpoolImpl     Threadpool 'default-handler-common': min=32, max=32, queue=1280
[2024-06-04 20:13:06.602] INFO    configserver     Container.com.yahoo.container.jdisc.ConfiguredApplication    Switching to the latest deployed set of configurations and components. Application config generation: 0
[2024-06-04 20:13:10.999] INFO    configproxy      configproxy.com.yahoo.vespa.config.JRTConnection     Connecting to tcp/vespa-0.vespa.danswer.svc.cluster.local:19070
[2024-06-04 20:13:11.101] INFO    configproxy      configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester    Request failed: Failed request (No application exists) from Connection { Socket[addr=/10.244.0.99,port=49258,localport=19070] }\nConnection spec: tcp/vespa-0.vespa.danswer.svc.cluster.local:19070

I tried with serveral configuration for

VESPA_CONFIGSERVERS=vespa-0.vespa.danswer.svc.cluster.local

But I couldn't start to work with Vespa in K8s.

[sd109]

I am encountering the same Vespa issue when trying to deploy Danswer, any suggestions would be appreciated.

[skeenan947]

Same issue here

[sd109]

I found the issue. Vespa k8s service labels don't match the labels applied to the Vespa pod by Danswer here. For now the fix is to add the following to your Danswer Helm chart values:

vespa:
  podLabels:
    app: vespa
    app.kubernetes.io/instance: danswer

but a more permanent fix would be to make the service labels take their value from {{ .Values.podLabels }} here.

[monchopena]

I changed the podLabels:

$ k describe po vespa-0

Name:         vespa-0
Namespace:    danswer
Priority:     0
Node:         kontabo-control-plane/172.18.0.2
Start Time:   Fri, 07 Jun 2024 18:48:43 +0200
Labels:       app=vespa
              app.kubernetes.io/instance=danswer

....

But the problem persists:

[2024-06-07 17:37:38.955] INFO    configproxy      configproxy.com.yahoo.vespa.config.proxy.DelayedResponseHandler      Timed out (timeout 25000) getting config name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local, will retry
[2024-06-07 17:37:45.377] INFO    config-sentinel  sentinel.config.frt.frtconfigagent   No response / error from config server. This is normal before an application package is deployed. (key: name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local) (errcode=103, validresponse:0), trying again in 7.000000 seconds
[2024-06-07 17:37:57.287] INFO    configproxy      configproxy.com.yahoo.vespa.config.proxy.RpcConfigSourceClient       Subscribe for 'name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local,aa9115ebf63cfc6721f75ada21d7cdfa' failed, closing subscriber: Subscribe for 'name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local' timed out (timeout was 44000 ms): name=cloud.config.sentinel,configId=hosts/vespa-0.vespa.danswer.svc.cluster.local, Current generation: 0, Generation changed: false, Config changed: false
[2024-06-07 17:37:57.291] INFO    configproxy      configproxy.com.yahoo.config.subscription.impl.JRTConfigRequester    Request failed: Failed request (No application exists) from Connection { Socket[addr=/10.244.0.182,port=59596,localport=19070] }\nConnection spec: tcp/vespa-0.vespa.danswer.svc.cluster.local:19070
---

I thinks this connection problem is not related with the podLabels.

[skeenan947]

Same here. I've tried swapping out multiple image tags as well, but I always get the same result.

[sd109]

Did you restart the Danswer pods after correcting the Vespa pod labels? Based on my reading of the various Danswer pod logs, one of the things the Danswer API server does on startup is connect to the Vespa service to set up the 'Application Package' (which your Vespa logs say is missing).

[monchopena]

I restarted the Danswer pods and now is running properly!
Thank you very much for your answer.