custom plugin - set an upstream as a http fallback server

[ryan4yin]

I want to write a custom plugin in lua, to configure an upstream as a http fallback server, the logic is:

• if the default upstream's response status_code != 2xx, then request the fallback server, and forward the fallback server's response back to the client.

I looked at the builtin plugin response-rewrite, batch-requests and proxy_mirror, but still not sure how to implement it.

beacuse I need to check the default upstream's response status_code, I think I should execute my fallback-plugin at _M.header_filter and _M.body_filter, the order maybe like this:

• _M.header_filter:
  • check the response code is not 2xx
  • request the fallback server using the client's arguments
  • set thengx.header and ngx.status to the fallback server's response headers & status_code
  • store the fallback server's response body into ctx.var
• _M.body_filter
  • set thengx.arg[1] to the fallback server's response headers & status_code

any suggestions? thanks~

[tzssangglass]

we have support this feature, like

Each node can be configured with a priority. A node with low priority will only be used when all the nodes with higher priority have been tried or are unavailable.

As the default priority is 0, nodes with negative priority can be configured as a backup.

ref: https://github.com/apache/apisix/blob/master/docs/en/latest/admin-api.md#request-body-parameters-3

[ryan4yin]

thanks, it works~

I resolve this by add proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504 http_429 non_idempotent; into APISIX config's configurationSnippet.httpSrv, and then set my backup server's priority to -1, every thing works fine~

But I have another problem now: APISIX's prometheus plguin seems do not have any metrics about retry, I can not monitor how frequently the retries happens through metrics.
It's not a big problem, because the access logs can records all the response_status in order, I just curious is it possible to records the retries metrics in prometheus plugin.

[tzssangglass]

Sounds like it makes sense, let's see if more people will need it, you can also create issues to describe this feature.

BTW, I think you've made clever use of retry and backup servers.

[ryan4yin]

in case anyone need this, I adjust the proxy-mirror's plugin-attr, set all the timeout too 300ms to avoid the original requests wait for the mirrored request to complete.

[tokers]

thanks, it works~

I resolve this by add proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504 http_429 non_idempotent; into APISIX config's configurationSnippet.httpSrv, and then set my backup server's priority to -1, every thing works fine~

But I have another problem now: APISIX's prometheus plguin seems do not have any metrics about retry, I can not monitor how frequently the retries happens through metrics. It's not a big problem, because the access logs can records all the response_status in order, I just curious is it possible to records the retries metrics in prometheus plugin.

That's a good user scenario. It's worth writing to the FAQ.

[ryan4yin]

and I customed the https://github.com/apache/apisix/blob/master/apisix/cli/ngx_tpl.lua to forward the X-Forward- prefixed headers to the mirror server.

I will create an issue latter to descibe this usage scenario and the problem I encountered, maybe this weekend.