pre-release WLO 1.5.0 - pods not scaling down when memory usage is less than target memory #787
Description
Test pre-release WebSphere Liberty operator 1.5.0, test with Daytrader10 application, deploy the application image to Amazon EKS with Kubernetes v1.33
% oc get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
ip-192-168-15-251.us-east-2.compute.internal Ready <none> 20h v1.33.3-eks-3abbec1 192.168.15.251 52.15.125.74 Amazon Linux 2023.8.20250818 6.12.40-63.114.amzn2023.x86_64 containerd://1.7.27
ip-192-168-66-240.us-east-2.compute.internal Ready <none> 20h v1.33.3-eks-3abbec1 192.168.66.240 3.15.204.211 Amazon Linux 2023.8.20250818 6.12.40-63.114.amzn2023.x86_64 containerd://1.7.27
Deployed app with the following settings
spec:
autoscaling:
minReplicas: 2
maxReplicas: 10
targetMemoryUtilizationPercentage: 80
behavior:
scaleUp:
tolerance: 0.02
scaleDown:
tolerance: 0.02
resources:
requests:
cpu: 500m
memory: 500Mi
limits:
cpu: 500m
memory: 1200Mi
Run 10 minutes stress test and monitor hpa during the test run
% oc -n daytrader10 get hpa -w
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
daytrader10 StatefulSet/daytrader10 memory: 51%/80% 2 10 2 2m19s
daytrader10 StatefulSet/daytrader10 memory: 51%/80% 2 10 2 2m30s
daytrader10 StatefulSet/daytrader10 memory: 57%/80% 2 10 2 2m45s
daytrader10 StatefulSet/daytrader10 memory: 55%/80% 2 10 2 3m
daytrader10 StatefulSet/daytrader10 memory: 56%/80% 2 10 2 3m15s
daytrader10 StatefulSet/daytrader10 memory: 56%/80% 2 10 2 3m30s
daytrader10 StatefulSet/daytrader10 memory: 56%/80% 2 10 2 3m45s
daytrader10 StatefulSet/daytrader10 memory: 56%/80% 2 10 2 4m
daytrader10 StatefulSet/daytrader10 memory: 56%/80% 2 10 2 4m15s
daytrader10 StatefulSet/daytrader10 memory: 57%/80% 2 10 2 4m30s
daytrader10 StatefulSet/daytrader10 memory: 57%/80% 2 10 2 4m45s
daytrader10 StatefulSet/daytrader10 memory: 62%/80% 2 10 2 5m
daytrader10 StatefulSet/daytrader10 memory: 66%/80% 2 10 2 5m15s
daytrader10 StatefulSet/daytrader10 memory: 66%/80% 2 10 2 5m30s
daytrader10 StatefulSet/daytrader10 memory: 69%/80% 2 10 2 5m45s
daytrader10 StatefulSet/daytrader10 memory: 68%/80% 2 10 2 6m
daytrader10 StatefulSet/daytrader10 memory: 71%/80% 2 10 2 6m16s
daytrader10 StatefulSet/daytrader10 memory: 73%/80% 2 10 2 6m31s
daytrader10 StatefulSet/daytrader10 memory: 74%/80% 2 10 2 6m46s
daytrader10 StatefulSet/daytrader10 memory: 74%/80% 2 10 2 7m1s
daytrader10 StatefulSet/daytrader10 memory: 74%/80% 2 10 2 7m16s
daytrader10 StatefulSet/daytrader10 memory: 76%/80% 2 10 2 7m31s
daytrader10 StatefulSet/daytrader10 memory: 77%/80% 2 10 2 7m46s
daytrader10 StatefulSet/daytrader10 memory: 77%/80% 2 10 2 8m1s
daytrader10 StatefulSet/daytrader10 memory: 78%/80% 2 10 2 8m16s
daytrader10 StatefulSet/daytrader10 memory: 79%/80% 2 10 2 8m31s
daytrader10 StatefulSet/daytrader10 memory: 79%/80% 2 10 2 8m46s
daytrader10 StatefulSet/daytrader10 memory: 80%/80% 2 10 2 9m1s
daytrader10 StatefulSet/daytrader10 memory: 81%/80% 2 10 2 9m16s
daytrader10 StatefulSet/daytrader10 memory: 82%/80% 2 10 2 9m31s
daytrader10 StatefulSet/daytrader10 memory: 83%/80% 2 10 2 9m46s
daytrader10 StatefulSet/daytrader10 memory: 83%/80% 2 10 2 10m
daytrader10 StatefulSet/daytrader10 memory: 85%/80% 2 10 2 10m
daytrader10 StatefulSet/daytrader10 memory: 85%/80% 2 10 2 10m
daytrader10 StatefulSet/daytrader10 memory: 86%/80% 2 10 2 10m
daytrader10 StatefulSet/daytrader10 memory: 87%/80% 2 10 2 11m
daytrader10 StatefulSet/daytrader10 memory: 87%/80% 2 10 2 11m
daytrader10 StatefulSet/daytrader10 memory: 90%/80% 2 10 2 11m
daytrader10 StatefulSet/daytrader10 memory: 88%/80% 2 10 3 11m
daytrader10 StatefulSet/daytrader10 memory: 67%/80% 2 10 3 12m
daytrader10 StatefulSet/daytrader10 memory: 73%/80% 2 10 3 12m
daytrader10 StatefulSet/daytrader10 memory: 76%/80% 2 10 3 12m
daytrader10 StatefulSet/daytrader10 memory: 78%/80% 2 10 3 12m
daytrader10 StatefulSet/daytrader10 memory: 78%/80% 2 10 3 13m
daytrader10 StatefulSet/daytrader10 memory: 82%/80% 2 10 3 13m
daytrader10 StatefulSet/daytrader10 memory: 83%/80% 2 10 3 13m
daytrader10 StatefulSet/daytrader10 memory: 86%/80% 2 10 3 13m
daytrader10 StatefulSet/daytrader10 memory: 85%/80% 2 10 3 14m
daytrader10 StatefulSet/daytrader10 memory: 86%/80% 2 10 3 14m
daytrader10 StatefulSet/daytrader10 memory: 86%/80% 2 10 3 14m
daytrader10 StatefulSet/daytrader10 memory: 87%/80% 2 10 3 14m
daytrader10 StatefulSet/daytrader10 memory: 87%/80% 2 10 3 15m
daytrader10 StatefulSet/daytrader10 memory: 89%/80% 2 10 3 15m
daytrader10 StatefulSet/daytrader10 memory: 89%/80% 2 10 4 15m
daytrader10 StatefulSet/daytrader10 memory: 73%/80% 2 10 4 15m
daytrader10 StatefulSet/daytrader10 memory: 75%/80% 2 10 4 16m
daytrader10 StatefulSet/daytrader10 memory: 77%/80% 2 10 4 16m
daytrader10 StatefulSet/daytrader10 memory: 78%/80% 2 10 4 16m
daytrader10 StatefulSet/daytrader10 memory: 83%/80% 2 10 4 16m
After the test done running and the memory usage is at 69%/80%, app pods is still at 4 and not scaling down to 2 (minReplicas)
daytrader10 StatefulSet/daytrader10 memory: 69%/80% 2 10 4 54m
daytrader10 StatefulSet/daytrader10 memory: 69%/80% 2 10 4 54m
daytrader10 StatefulSet/daytrader10 memory: 69%/80% 2 10 4 54m
daytrader10 StatefulSet/daytrader10 memory: 69%/80% 2 10 4 55m
..........
daytrader10 StatefulSet/daytrader10 memory: 68%/80% 2 10 4 65m
daytrader10 StatefulSet/daytrader10 memory: 68%/80% 2 10 4 66m