APISIX returns 502 when using Consul

[maybgit]

Apisix uses consul service discovery. Services are dynamic ports. Every time a service is built, it will basically be 502, unless apisix is restarted.
Reasons for guessing:
After the microservice is restarted, the port in the consul kv configuration is changed, and apisix saves the kv configuration that was pulled last time, so the result is always 502
After restarting the apisix service, it is normal. After restarting apisix, the configuration is pulled from consul kv again
Is there any way to solve this problem?

[juzhiyuan]

Hi @tzssangglass, could you please give some hints on this issue? 😄

[WeCreos]

Re-read the thread again,,,☺️

[maybgit]

consul_kv:
keepalive: false
fetch_interval: 3

By setting this, you can pull the configuration from consuk kv regularly, it will not be 502, but according to the document, the official does not recommend setting this, I don’t know if there is any other solution

[tzssangglass]

I can't think of any good options.

微服务重启后

I think it's normal that restart means the upstream service is temporarily unavailable.

[maybgit]

你审题有问题

[tzssangglass]

Dynamic port or not, wouldn't the service not be available during the service restart?

你审题有问题

Don't just say I'm not right. Tell me what's wrong, don't make me guess.

[juzhiyuan]

Hi @maybgit, please use English in the public channels, for other volunteers cannot understand what you mean.

[juzhiyuan]

Dynamic port or not, wouldn't the service not be available during the service restart?

你审题有问题

Don't just say I'm not right. Tell me what's wrong, don't make me guess.

And please provide or give detailed feedbacks or issues what you meet, or we don't know how to help you.

[tzssangglass]

The fetch_interval controls the frequency of the pull data from consul. But there will always be a cache, and if the service restarts during the cache time, it will still return 502.

[maybgit]

apisix first get
[
"upstreams/admin-api/192.168.1.234:20604",
"upstreams/admin-api/192.168.1.235:40033",
"upstreams/admin-api/192.168.1.236:32598"
]
After the service restarts, apisix no longer obtains configuration from consul kv，unless restart apisix.
[
"upstreams/admin-api/192.168.1.234:30467",
"upstreams/admin-api/192.168.1.235:32022",
"upstreams/admin-api/192.168.1.236:23172"
]

There is a solution, set keepalive and fetch_interval timing pull, but the official does not recommend this, I don’t know if there is any other solution

[tzssangglass]

When the service restart is complete, notify APISIX to force refresh data from consul.

APISIX can expose the interface to force refresh, but it is not supported yet.

[spacewander]

When keepalive is true (which is the default), APISIX will use HTTP long poll to get the consul KV data. Once the consul KV data is changed, APISIX is expected to get a HTTP response from consul and update the services list.

Could you add more log (or decrease the log level) in

apisix/apisix/discovery/consul_kv/init.lua

Line 240 in 50af9bc

function _M.connect(premature, consul_server, retry_delay)

to see why the update is not taking effect immediately? Also, we need some HTTP packet capture files to see which part failed to deliver the latest data. (Consul doesn't give the HTTP response, or APISIX failed to get the update).

[WeCreos]

Same here