APISIX 100% Memory Usage on RedHat Openshift Kubernetes Cluster

[geeky-akshay]

Hi Community,

I'm seeing high memory usage of apisix-data-plane pod on RedHat Openshift Cluster. After periodic requests the usage climbs up to 100%. Where as on AWS EKS cluster the memory usage is significantly low.

On RedHat Openshift Cluster, the memory limit is set to 3Gi and it is using 100% of the memory.

$ oc adm top pods -n apisix
NAME                                        CPU(cores)   MEMORY(bytes)
apisix-control-plane-98ccc95b8-v2lgl        154m         2530Mi
apisix-dashboard-5ff849bcc8-fsh9j           15m          47Mi
apisix-data-plane-75cfc58f96-2chzv          288m         3071Mi

$ oc describe deployment apisix-data-plane | grep -i limits -A 3
    Limits:
      cpu:                1500m
      ephemeral-storage:  2Gi
      memory:             3Gi

Where as on AWS EKS Cluster

$ kubectl top pod -n apisix
NAME                                        CPU(cores)   MEMORY(bytes)
apisix-control-plane-f788744f-zqdfg         37m          339Mi
apisix-dashboard-5d98674fc4-rxnq2           67m          52Mi
apisix-data-plane-558498cf46-q25mc          22m          373Mi

APISIX VERSION: 3.12.0

$ apisix version
/opt/bitnami/apisix/openresty//luajit/bin/luajit /usr/local/apisix/apisix/cli/apisix.lua version
3.12.0

LUAROCKS VERSION: 3.8.0

$ luarocks --version
/opt/bitnami/apisix/openresty/bin/luarocks 3.8.0
LuaRocks main command-line interface

Can anyone help me identify the issue?

Thanks

[juzhiyuan]

Hi @geeky-akshay , to debug the root reason, we need a relroducable env like yours.

1. Your configurations (Gateway and Services/Plugins)
2. Your deployment.

Can you take some time to orgainze and send it to us? so we can have a try

[geeky-akshay]

Environment

RedHat Openshift Container Platform Version: 4.16.36

$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.16.36   True        False         13d     Cluster version is 4.16.36

LoadBalancer: MetalLB
MetalLB is installed using metallb operator (https://docs.redhat.com/en/documentation/openshift_container_platform/4.9/html/networking/load-balancing-with-metallb#metallb-operator-install)
You'll have to create necessary L2Advertisemet and IPAddressPool which will be used by APISIX for External IP address

Monitoring Stack: https://docs.redhat.com/en/documentation/openshift_container_platform/3.11/html/configuring_clusters/prometheus-cluster-monitoring#installing-monitoring-stack

Deployment

APISIX is deployed using Bitnami Helm Chart: https://github.com/bitnami/charts/tree/main/bitnami/apisix (version: 4.2.5, appVersion: 3.12.0)

Custom Values:

global:
  security:
    allowInsecureImages: true

controlPlane:
  extraConfig:
    apisix:
      ssl:
        ssl_trusted_certificate: '/bitnami/certs/{{ .Values.controlPlane.tls.certCAFilename }}'
  lifecycleHooks:
    postStart:
      exec:
        command:
          - /bin/sh
          - -c
          - |
            sleep 5;
            rm /usr/local/apisix/logs/worker_events.sock
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      labels:
        clusterobservability: "1"
        cluster_component: "apisix-control-plane"
      metricRelabelings:
      - targetLabel: "cluster_component"
        replacement: "apisix-control-plane"
  resourcesPreset: "large"
dataPlane:
  extraConfig:
    apisix:
      ssl:
        fallback_sni: "apisix-data-plane.local"
    plugin_attr:
      prometheus:
        metrics:
          http_status:
            expire: 600
          http_latency:
            expire: 600
          bandwidth:
            expire: 600
          upstream_status:
            expire: 600
      redirect:
        https_port: 443
  lifecycleHooks:
    postStart:
      exec:
        command:
          - /bin/sh
          - -c
          - |
            sleep 5;
            rm /usr/local/apisix/logs/worker_events.sock
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      labels:
        clusterobservability: "1"
        cluster_component: "apisix-data-plane"
      metricRelabelings:
      - targetLabel: "cluster_component"
        replacement: "apisix-data-plane"
  resourcesPreset: "large"
ingressController:
  extraConfig:
    apisix_resource_sync_interval: 30m
    kubernetes:
      resync_interval: 1h
  metrics:
    enabled: true
    serviceMonitor:
      enabled: true
      labels:
        clusterobservability: "1"
        cluster_component: "apisix-ingress-controller"
      metricRelabelings:
      - targetLabel: "cluster_component"
        replacement: "apisix-ingress-controller"
  replicaCount: 3
  resourcesPreset: "small"
etcd:
  autoCompactionMode: "revision"
  autoCompactionRetention: "3"
  extraEnvVars:
    - name: ETCD_QUOTA_BACKEND_BYTES
      value: "4294967296"
  metrics:
    enabled: true
    podMonitor:
      enabled: true
      namespace: "openshift-monitoring"
      additionalLabels:
        clusterobservability: "1"
        cluster_component: "apisix-etcd"
      relabelings:
      - targetLabel: "cluster_component"
        replacement: "apisix-etcd"
  resourcesPreset: null
  resources:
    requests:
      cpu: 500m
      memory: 512Mi

Extra Resources:
ApisixClusterConfig.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixClusterConfig
metadata:
  name: default
  namespace: apisix
spec:
  monitoring:
    prometheus:
      enable: true

ApisixGlobalRule.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
  name: global
  namespace: apisix
spec:
  plugins:
    - name: redirect
      enable: true
      config:
        http_to_https: true

ApisixTls.yaml

apiVersion: apisix.apache.org/v2
kind: ApisixTls
metadata:
  name: apisix-tls
  namespace: apisix
spec:
  hosts:
  - <APISIX-EXTERNAL-IP-ADDRESS>
  - apisix-data-plane.local
  secret:
    name: <CERT-CONTAINING-APISIX-EXTERNAL-IP-ADDRESS>
    namespace: apisix

[geeky-akshay]

Upon more investigation I found that, when worker_processes is set to auto, nginx creates different number of workers in AWS that on Openshift.

In AWS I see 4 worker processes, hence memory usage is lower

$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
apisix         1       0  0 May19 ?        00:00:00 nginx: master process openresty -p /usr/local/apisix -g daemon off;
apisix        16       1  0 May19 ?        00:00:52 nginx: worker process
apisix        17       1  0 May19 ?        00:00:50 nginx: worker process
apisix        18       1  0 May19 ?        00:00:49 nginx: worker process
apisix        19       1  0 May19 ?        00:00:50 nginx: worker process
apisix        20       1  0 May19 ?        00:00:00 nginx: cache manager process
apisix        22       1  0 May19 ?        00:01:49 nginx: privileged agent process
apisix        29       0  0 06:47 pts/0    00:00:00 sh
apisix        42      29  0 06:58 pts/0    00:00:00 ps -ef
$

In Openshift I see 80 worker processes, hence memory usage is higher

apisix@apisix-data-plane-669f775f9c-t5n9k:/$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
apisix         1       0  0 May19 ?        00:00:00 nginx: master process openresty -p /usr/local/apisix -g daemon off;
apisix        14       1  0 May19 ?        00:00:35 nginx: worker process
apisix        15       1  0 May19 ?        00:00:21 nginx: worker process
apisix        16       1  0 May19 ?        00:03:51 nginx: worker process
apisix        17       1  0 May19 ?        00:00:21 nginx: worker process
apisix        18       1  0 May19 ?        00:03:51 nginx: worker process
apisix        19       1  0 May19 ?        00:00:20 nginx: worker process
apisix        20       1  0 May19 ?        00:00:21 nginx: worker process
apisix        21       1  0 May19 ?        00:00:21 nginx: worker process
apisix        22       1  0 May19 ?        00:00:21 nginx: worker process
apisix        23       1  0 May19 ?        00:00:20 nginx: worker process
apisix        26       1  0 May19 ?        00:00:21 nginx: worker process
apisix        28       1  0 May19 ?        00:03:51 nginx: worker process
apisix        29       1  0 May19 ?        00:03:51 nginx: worker process
apisix        31       1  0 May19 ?        00:00:21 nginx: worker process
apisix        33       1  0 May19 ?        00:03:51 nginx: worker process
apisix        36       1  0 May19 ?        00:03:51 nginx: worker process
apisix        37       1  0 May19 ?        00:00:21 nginx: worker process
apisix        40       1  0 May19 ?        00:00:21 nginx: worker process
apisix        42       1  0 May19 ?        00:03:53 nginx: worker process
apisix        45       1  0 May19 ?        00:00:20 nginx: worker process
apisix        46       1  0 May19 ?        00:03:51 nginx: worker process
apisix        49       1  0 May19 ?        00:03:51 nginx: worker process
apisix        51       1  0 May19 ?        00:00:21 nginx: worker process
apisix        53       1  0 May19 ?        00:03:51 nginx: worker process
apisix        55       1  0 May19 ?        00:03:50 nginx: worker process
apisix        58       1  0 May19 ?        00:03:51 nginx: worker process
apisix        59       1  0 May19 ?        00:03:51 nginx: worker process
apisix        60       1  0 May19 ?        00:03:50 nginx: worker process
apisix        61       1  0 May19 ?        00:03:51 nginx: worker process
apisix        63       1  0 May19 ?        00:00:21 nginx: worker process
apisix        65       1  0 May19 ?        00:03:51 nginx: worker process
apisix        67       1  0 May19 ?        00:03:51 nginx: worker process
apisix        71       1  0 May19 ?        00:00:20 nginx: worker process
apisix        74       1  0 May19 ?        00:00:21 nginx: worker process
apisix        76       1  0 May19 ?        00:00:21 nginx: worker process
apisix        77       1  0 May19 ?        00:00:21 nginx: worker process
apisix        78       1  0 May19 ?        00:00:20 nginx: worker process
apisix        79       1  0 May19 ?        00:00:21 nginx: worker process
apisix        82       1  0 May19 ?        00:03:51 nginx: worker process
apisix        83       1  0 May19 ?        00:03:51 nginx: worker process
apisix        84       1  0 May19 ?        00:00:21 nginx: worker process
apisix        85       1  0 May19 ?        00:03:51 nginx: worker process
apisix        86       1  0 May19 ?        00:03:51 nginx: worker process
apisix        87       1  0 May19 ?        00:00:21 nginx: worker process
apisix        88       1  0 May19 ?        00:03:51 nginx: worker process
apisix        89       1  0 May19 ?        00:03:51 nginx: worker process
apisix        90       1  0 May19 ?        00:03:51 nginx: worker process
apisix        91       1  0 May19 ?        00:00:21 nginx: worker process
apisix        92       1  0 May19 ?        00:00:21 nginx: worker process
apisix        93       1  0 May19 ?        00:00:21 nginx: worker process
apisix        94       1  0 May19 ?        00:03:51 nginx: worker process
apisix        96       1  0 May19 ?        00:03:52 nginx: worker process
apisix        97       1  0 May19 ?        00:03:51 nginx: worker process
apisix       101       1  0 May19 ?        00:00:21 nginx: worker process
apisix       107       1  0 May19 ?        00:00:21 nginx: worker process
apisix       108       1  0 May19 ?        00:03:50 nginx: worker process
apisix       109       1  0 May19 ?        00:00:20 nginx: worker process
apisix       110       1  0 May19 ?        00:03:51 nginx: worker process
apisix       111       1  0 May19 ?        00:03:51 nginx: worker process
apisix       112       1  0 May19 ?        00:03:51 nginx: worker process
apisix       115       1  0 May19 ?        00:00:20 nginx: worker process
apisix       116       1  0 May19 ?        00:00:21 nginx: worker process
apisix       119       1  0 May19 ?        00:03:51 nginx: worker process
apisix       120       1  0 May19 ?        00:00:21 nginx: worker process
apisix       121       1  0 May19 ?        00:00:20 nginx: worker process
apisix       129       1  0 May19 ?        00:03:50 nginx: worker process
apisix       130       1  0 May19 ?        00:03:50 nginx: worker process
apisix       132       1  0 May19 ?        00:03:50 nginx: worker process
apisix       136       1  0 May19 ?        00:00:21 nginx: worker process
apisix       137       1  0 May19 ?        00:03:50 nginx: worker process
apisix       138       1  0 May19 ?        00:03:50 nginx: worker process
apisix       139       1  0 May19 ?        00:00:21 nginx: worker process
apisix       140       1  0 May19 ?        00:00:21 nginx: worker process
apisix       141       1  0 May19 ?        00:03:50 nginx: worker process
apisix       142       1  0 May19 ?        00:03:50 nginx: worker process
apisix       150       1  0 May19 ?        00:00:21 nginx: worker process
apisix       151       1  0 May19 ?        00:03:50 nginx: worker process
apisix       152       1  0 May19 ?        00:00:21 nginx: worker process
apisix       153       1  0 May19 ?        00:03:51 nginx: worker process
apisix       154       1  0 May19 ?        00:00:21 nginx: worker process
apisix       155       1  0 May19 ?        00:00:00 nginx: cache manager process
apisix       158       1  0 May19 ?        00:04:04 nginx: privileged agent process
apisix       221       0  0 06:22 pts/0    00:00:00 bash
apisix       228     221  0 06:22 pts/0    00:00:00 ps -ef

When I explicitly set worker_processes: 4 the memory usage is reduced in Openshift Cluster

$ oc adm top pod apisix-data-plane-669f775f9c-tpnnn
NAME                                 CPU(cores)   MEMORY(bytes)
apisix-data-plane-669f775f9c-tpnnn   3m           87Mi

Is there a way to control nginx worker processes in auto mode so that it doesn't run out of memory?

[geeky-akshay]

More investigation:
nginx looks at nproc and runs number of worker processes.

In AWS, nproc is 4 hence worker processes are 4.

apisix@apisix-data-plane-855f468c49-lpvq4:/$ nproc
4
apisix@apisix-data-plane-855f468c49-lpvq4:/$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
apisix         1       0  0 May19 ?        00:00:00 nginx: master process openresty -p /usr/local/apisix -g daemon off;
apisix        16       1  0 May19 ?        00:00:56 nginx: worker process
apisix        17       1  0 May19 ?        00:00:53 nginx: worker process
apisix        18       1  0 May19 ?        00:00:53 nginx: worker process
apisix        19       1  0 May19 ?        00:00:53 nginx: worker process
apisix        20       1  0 May19 ?        00:00:00 nginx: cache manager process
apisix        22       1  0 May19 ?        00:01:56 nginx: privileged agent process
apisix        43       0  0 08:13 pts/0    00:00:00 sh
apisix        50      43  0 08:13 pts/0    00:00:00 bash
apisix        52      50  0 08:13 pts/0    00:00:00 ps -ef

Whereas in Openshift nproc is 80 hence I see worker processes as 80

1001350000@apisix-data-plane-797d56599b-7hz2g:/$ nproc
80
1001350000@apisix-data-plane-797d56599b-7hz2g:/$ ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
1001350+       1       0  0 May19 ?        00:00:00 nginx: master process openresty -p /usr/local/apisix -g daemon off;
1001350+      14       1  0 May19 ?        00:04:35 nginx: worker process
1001350+      15       1  0 May19 ?        00:00:16 nginx: worker process
1001350+      16       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      17       1  0 May19 ?        00:00:17 nginx: worker process
1001350+      18       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      19       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      20       1  0 May19 ?        00:04:29 nginx: worker process
1001350+      21       1  0 May19 ?        00:00:16 nginx: worker process
1001350+      22       1  0 May19 ?        00:04:29 nginx: worker process
1001350+      23       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      25       1  0 May19 ?        00:04:32 nginx: worker process
1001350+      27       1  0 May19 ?        00:00:16 nginx: worker process
1001350+      29       1  0 May19 ?        00:04:29 nginx: worker process
1001350+      31       1  0 May19 ?        00:04:32 nginx: worker process
1001350+      34       1  0 May19 ?        00:00:15 nginx: worker process
1001350+      35       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      38       1  0 May19 ?        00:04:34 nginx: worker process
1001350+      40       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      43       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      45       1  0 May19 ?        00:04:32 nginx: worker process
1001350+      47       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      49       1  0 May19 ?        00:00:17 nginx: worker process
1001350+      51       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      53       1  0 May19 ?        00:04:28 nginx: worker process
1001350+      55       1  0 May19 ?        00:00:15 nginx: worker process
1001350+      57       1  0 May19 ?        00:04:29 nginx: worker process
1001350+      60       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      61       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      62       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      64       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      65       1  0 May19 ?        00:04:29 nginx: worker process
1001350+      66       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      69       1  0 May19 ?        00:04:34 nginx: worker process
1001350+      71       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      72       1  0 May19 ?        00:00:15 nginx: worker process
1001350+      73       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      74       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      80       1  0 May19 ?        00:04:32 nginx: worker process
1001350+      83       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      85       1  0 May19 ?        00:04:33 nginx: worker process
1001350+      86       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      87       1  0 May19 ?        00:00:16 nginx: worker process
1001350+      88       1  0 May19 ?        00:04:32 nginx: worker process
1001350+      89       1  0 May19 ?        00:04:28 nginx: worker process
1001350+      92       1  0 May19 ?        00:04:31 nginx: worker process
1001350+      94       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      95       1  0 May19 ?        00:00:15 nginx: worker process
1001350+      96       1  0 May19 ?        00:00:14 nginx: worker process
1001350+      97       1  0 May19 ?        00:04:33 nginx: worker process
1001350+      98       1  0 May19 ?        00:04:30 nginx: worker process
1001350+      99       1  0 May19 ?        00:04:32 nginx: worker process
1001350+     100       1  0 May19 ?        00:00:15 nginx: worker process
1001350+     101       1  0 May19 ?        00:04:32 nginx: worker process
1001350+     103       1  0 May19 ?        00:04:29 nginx: worker process
1001350+     105       1  0 May19 ?        00:04:31 nginx: worker process
1001350+     106       1  0 May19 ?        00:04:31 nginx: worker process
1001350+     107       1  0 May19 ?        00:04:29 nginx: worker process
1001350+     108       1  0 May19 ?        00:04:30 nginx: worker process
1001350+     109       1  0 May19 ?        00:04:31 nginx: worker process
1001350+     111       1  0 May19 ?        00:04:29 nginx: worker process
1001350+     112       1  0 May19 ?        00:04:27 nginx: worker process
1001350+     113       1  0 May19 ?        00:04:30 nginx: worker process
1001350+     114       1  0 May19 ?        00:04:30 nginx: worker process
1001350+     115       1  0 May19 ?        00:04:30 nginx: worker process
1001350+     116       1  0 May19 ?        00:04:29 nginx: worker process
1001350+     117       1  0 May19 ?        00:04:33 nginx: worker process
1001350+     118       1  0 May19 ?        00:04:28 nginx: worker process
1001350+     119       1  0 May19 ?        00:00:15 nginx: worker process
1001350+     120       1  0 May19 ?        00:00:17 nginx: worker process
1001350+     122       1  0 May19 ?        00:04:30 nginx: worker process
1001350+     123       1  0 May19 ?        00:04:31 nginx: worker process
1001350+     125       1  0 May19 ?        00:04:32 nginx: worker process
1001350+     126       1  0 May19 ?        00:00:15 nginx: worker process
1001350+     127       1  0 May19 ?        00:04:32 nginx: worker process
1001350+     128       1  0 May19 ?        00:04:30 nginx: worker process
1001350+     134       1  0 May19 ?        00:04:28 nginx: worker process
1001350+     136       1  0 May19 ?        00:04:29 nginx: worker process
1001350+     137       1  0 May19 ?        00:04:32 nginx: worker process
1001350+     138       1  0 May19 ?        00:00:14 nginx: worker process
1001350+     139       1  0 May19 ?        00:04:31 nginx: worker process
1001350+     141       1  0 May19 ?        00:00:00 nginx: cache manager process
1001350+     143       1  0 May19 ?        00:05:01 nginx: privileged agent process
1001350+     191       0  0 07:32 pts/0    00:00:00 bash
1001350+     199       0  0 08:06 pts/1    00:00:00 bash
1001350+     214     199  0 08:13 pts/1    00:00:00 ps -ef

[geeky-akshay]

I guess setting worker_processes depending upon number of cpu allocated to container would be the solution.