proxy-write and grpc-web not work together

[wyudong]

I'm trying to make two routes and two upstream grpc service work like this:

route    upstream uri   upstream name
/ac/*    /*             service-ac
/po/*    /*             service-po

Since I saw this issue said the route path should be consistent with the uri provided by the backend service, I added the proxy-rewrite plugin to my routes conf. But the web client got empty response from the server and grpc-message: unknown service ac/ac.foo appearred in the response header.

The upstream service is configured correctly because if I remove the proxy-rewrite plugin and use /* as one of the route path, e.g. /* -> /* (service-ac), the client communicates with service-ac correctly. But setting /* to service-po again does not work. I guess the routes have conflict when both setting to /*.

Am I doing anything wrong when dealing with the proxy-rewrite and grpc-web? And what's the best practice to proxy the grpc service which is hosted at the root domain?

{
  "uri": "/ac/*",
  "name": "Ac",
  "methods": [
    "POST",
    "OPTIONS"
  ],
  "plugins": {
    "grpc-web": {
      "disable": false
    },
    "proxy-rewrite": {
      "regex_uri": [
        "^/ac/(.*)",
        "/$1"
      ]
    }
  },
  "upstream_id": "433572647238894232",
  "labels": {
    "API_VERSION": "v1"
  },
  "status": 1
}
{
  "uri": "/po/*",
  "name": "Po",
  "methods": [
    "POST",
    "OPTIONS"
  ],
  "plugins": {
    "grpc-web": {
      "disable": false
    },
    "proxy-rewrite": {
      "regex_uri": [
        "^/po/(.*)",
        "/$1"
      ]
    }
  },
  "upstream_id": "433579894627107480",
  "labels": {
    "API_VERSION": "v1"
  },
  "status": 1
}

Environments

I installed apisix and apisix-dashboard using helm.

helm install apisix apisix/apisix -n apisix --set gateway.type=NodePort 
helm install apisix-dashboard apisix/apisix-dashboard -n apisix

The apisix ingress conf:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/cors-allow-headers: X-User-Agent,Content-Type,X-GRPC-WEB
    nginx.ingress.kubernetes.io/cors-allow-methods: OPTIONS,GET,POST,PUT,DELETE
    nginx.ingress.kubernetes.io/cors-allow-origin: '*'
    nginx.ingress.kubernetes.io/enable-cors: "true"
  name: example
  namespace: apisix
spec:
  rules:
  - host: api6.example.com
    http:
      paths:
      - backend:
          service:
            name: apisix-gateway
            port:
              number: 80
        pathType: ImplementationSpecific
  tls:
  - hosts:
    - api6.example.com
    secretName: apisix-tls
status:
  loadBalancer:
    ingress:
    - ip: 192.168.66.77

[tzssangglass]

PLAT at: https://github.com/apache/apisix/blob/master/docs/en/latest/plugins/grpc-web.md#enabling-the-plugin

While using the grpc-web Plugin, always using a prefix matching pattern (/, /grpc/example/) for matching Routes. This is because the gRPC Web client passes the package name, the service interface name, the method name and other information in the proto in the URI. For example, /path/a6.RouteService/Insert.

So, when absolute matching is used, the Plugin would not be hit and the information from the proto would not be extracted.

for example,

apisix/t/plugin/grpc-web/a6/route.proto

Lines 20 to 25 in 4c75c26

	package a6;
	option go_package = "./;a6";
	service RouteService {
	rpc GetRoute(Query) returns (Route) {}

if you want to access GetRoute, the path should be /a6.RouteService/GetRoute, and config like

curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "uri":"/grpc/web/*",
    "plugins":{
        "grpc-web":{}
    },
    "upstream":{
        "scheme":"grpc",
        "type":"roundrobin",
        "nodes":{
            "127.0.0.1:1980":1
        }
    }
}'

access this route by js, the request path maybe like http://127.0.0.1:1984/grpc/web/a6.RouteService/GetRoute

the path here

apisix/apisix/plugins/grpc-web.lua

Lines 94 to 99 in 4c75c26

	local path = ctx.curr_req_matched[":ext"]
	if path:byte(1) ~= core.string.byte("/") then
	path = "/" .. path
	end
	req_set_uri(path)

would be /a6.RouteService/GetRoute.

this would work.

in your way, add a route config like "uri":"/*" would be failed, this has nothing to do with proxy-rewrite, grpc-web currently only supports the above usage.

[tzssangglass]

• route config like "uri":"/grpc/web/*", the ctx.curr_req_matched[":ext"] is a6.RouteService/GetRoute
• route config like "uri":"/*", the ctx.curr_req_matched[":ext"] is grpc/web/a6.RouteService/GetRoute

[wyudong]

@tzssangglass Thanks for your explanation.

After some tries, I find the grpc-web plugin actually works if apisix is installed by docker. But the helm version does not work. Route uri like /foo/* will fail to find the upstream service. Do you have any idea to locate the error?

I make the request from web client but the reponse returns 200 with a header said grpc-message: unknown service. I'm wondering the reason why the docker and helm version behaves differently. The upstream configs are same btw.

[tzssangglass]

I'm wondering the reason why the docker and helm version behaves differently. The upstream configs are same btw.

The behavior of APISIX running in helm and docker should be the same. I guess it's your configuration difference that's causing this.
BTW, the APISIX installed in helm is 2.15 by default, what is the version number of the APISIX image you installed via docker?

[wyudong]

BTW, the APISIX installed in helm is 2.15 by default, what is the version number of the APISIX image you installed via docker?

Docker version is 2.99.0 for my local and 3.0.0 for the server. Maybe v3 has something different?
Helm is 2.15.

[wyudong]

The discussion is about the helm version of apisix.

Removing the second slash from the route path works.

route   upstream name   result
/ac/*   service-ac      ❎
/ac*    service-ac      ✅

Moreover, even the grpc-web plugin is disabled, it makes no difference to the result. The grpc-web is not found in this list: https://github.com/apache/apisix-helm-chart/blob/master/charts/apisix/values.yaml#L217

"plugins": {
  "grpc-web": {
    "disable": true
  }
},

[tzssangglass]

Moreover, even the grpc-web plugin is disabled, it makes no difference to the result. The grpc-web is not found in this list: apache/apisix-helm-chart@master/charts/apisix/values.yaml#L217

I'm not sure. maybe you can submit an issue on apisix-helm-chart to sak why disable grpc-web plugin by default.

[tzssangglass]

I think using the grpc-web plugin without proxy-rewrite to modify the uri is the right way to use it.

If the same configuration produces inconsistent behavior in helm and docker, please give detailed steps to reproduce them so I can reproduce them locally.

[wyudong]

Yes, proxy-rewrite is not used now.

If the same configuration produces inconsistent behavior in helm and docker, please give detailed steps to reproduce them so I can reproduce them locally.

I will write the steps later. Thanks.