NDM using extremely high memory #683
Description
What steps did you take and what happened:
I have a 64 GB server as part of my kubernetes cluster. I'm running openebs on the cluster. NDM is running on all nodes, but on this node, it's eating up 50% of the total memory. Examining the logs shows that it seems to be stuck in some kind of loop
This just repeats over and over.
What did you expect to happen:
On my other servers, ndm is reported as taking up <0.01% of memory, which is what I expect here
The output of the following commands will help us better understand what's going on: https://gist.github.com/tathougies/70f8cb54929ea890c07896e0b48616ff
kubectl get pods -n openebskubectl get blockdevices -n openebs -o yamlkubectl get blockdeviceclaims -n openebs -o yamlkubectl logs <ndm daemon pod name> -n openebslsblkfrom nodes where ndm daemonset is running
Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]
Environment:
- OpenEBS version: 2.12.1
- Kubernetes version (use
kubectl version): version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.3", GitCommit:"c92036820499fedefec0f847e2054d824aea6cd1", GitTreeState:"archive", BuildDate:"1980-01-01T00:00:00Z", GoVersion:"go1.16.13", Compiler:"gc", Platform:"linux/amd64"} - Kubernetes installer & version: Using stock kubernetes NixOS
- Cloud provider or hardware configuration: Custom
- Type of disks connected to the nodes (eg: Virtual Disks, GCE/EBS Volumes, Physical drives etc): 2.5" SAS SSDs connected to RAID controller in pass-thru mode
- OS (e.g. from
/etc/os-release):
NAME=NixOS
ID=nixos
VERSION="21.11 (Porcupine)"
VERSION_CODENAME=porcupine
VERSION_ID="21.11"
BUILD_ID="21.11.336386.bacbfd713b4"
PRETTY_NAME="NixOS 21.11 (Porcupine)"
LOGO="nix-snowflake"
HOME_URL="https://nixos.org/"
DOCUMENTATION_URL="https://nixos.org/learn.html"
SUPPORT_URL="https://nixos.org/community.html"
BUG_REPORT_URL="https://github.com/NixOS/nixpkgs/issues"