shkeeper stopped working

[anycolo]

How can i even debug this?

[github-actions]

Official Statement from the SHKeeper Development Team

We would like to clarify that the SHKeeper development team consists solely of the following:

• @vsyshost
• @dmytro-samoylenko
• @Olxndr
• @yukonet

It is managed through our official GitHub account https://github.com/vsys-host

We NEVER:

• Share or redirect users to unofficial or suspicious links.
• Request private keys, funds, or other sensitive information.

If you encounter any comments, issues, or users claiming to represent SHKeeper and sharing external links or providing questionable support, please report them immediately. These individuals are not affiliated with the SHKeeper team.

For accurate information and official support, please refer only to:

Our GitHub account: https://github.com/vsys-host/shkeeper.io
Our official SHKeeper website: https://shkeeper.io/
Our official VSYS Host website: https://vsys.host/

Thank you for your understanding and vigilance.

SHKeeper Development Team

[yukonet]

Hello

Please check if there is enough free disk space on your server.

You can check the logs of a specific pod with the command

kubectl -n shkeeper logs <pod_name>

You can also use the command to get more information about a pod.

kubectl -n shkeeper describe pod <pod_name>

You can see the latest events with the command

kubectl get events -A

[anycolo]

There is 100GB free space..

[anycolo]

kubectl get events -A
NAMESPACE      LAST SEEN   TYPE      REASON                 OBJECT                                         MESSAGE
kube-system    57m         Warning   FailedDaemonPod        daemonset/svclb-shkeeper-external-2774e482     Found failed daemon pod kube-system/svclb-shkeeper-external-2774e482-hwvv9 on node shkeeper.anycolo.net, will try to kill it
kube-system    57m         Normal    SuccessfulDelete       daemonset/svclb-shkeeper-external-2774e482     Deleted pod: svclb-shkeeper-external-2774e482-hwvv9
kube-system    57m         Warning   FailedDaemonPod        daemonset/svclb-traefik-8dbd47f2               Found failed daemon pod kube-system/svclb-traefik-8dbd47f2-8b6dp on node shkeeper.anycolo.net, will try to kill it
kube-system    57m         Normal    SuccessfulDelete       daemonset/svclb-traefik-8dbd47f2               Deleted pod: svclb-traefik-8dbd47f2-8b6dp
kube-system    57m         Normal    SuccessfulCreate       daemonset/svclb-shkeeper-external-2774e482     Created pod: svclb-shkeeper-external-2774e482-prvfk
kube-system    57m         Normal    SuccessfulCreate       daemonset/svclb-traefik-8dbd47f2               Created pod: svclb-traefik-8dbd47f2-72ltr
kube-system    57m         Normal    Scheduled              pod/svclb-shkeeper-external-2774e482-prvfk     Successfully assigned kube-system/svclb-shkeeper-external-2774e482-prvfk to shkeeper.anycolo.net
kube-system    57m         Warning   Evicted                pod/svclb-shkeeper-external-2774e482-prvfk     The node had condition: [DiskPressure].
kube-system    57m         Normal    Scheduled              pod/svclb-traefik-8dbd47f2-72ltr               Successfully assigned kube-system/svclb-traefik-8dbd47f2-72ltr to shkeeper.anycolo.net
kube-system    57m         Warning   Evicted                pod/svclb-traefik-8dbd47f2-72ltr               The node had condition: [DiskPressure].
kube-system    42m         Warning   FailedDaemonPod        daemonset/svclb-shkeeper-external-2774e482     Found failed daemon pod kube-system/svclb-shkeeper-external-2774e482-prvfk on node shkeeper.anycolo.net, will try to kill it
kube-system    42m         Normal    SuccessfulDelete       daemonset/svclb-shkeeper-external-2774e482     Deleted pod: svclb-shkeeper-external-2774e482-prvfk
kube-system    42m         Warning   FailedDaemonPod        daemonset/svclb-traefik-8dbd47f2               Found failed daemon pod kube-system/svclb-traefik-8dbd47f2-72ltr on node shkeeper.anycolo.net, will try to kill it
kube-system    42m         Normal    SuccessfulDelete       daemonset/svclb-traefik-8dbd47f2               Deleted pod: svclb-traefik-8dbd47f2-72ltr
kube-system    42m         Normal    SuccessfulCreate       daemonset/svclb-shkeeper-external-2774e482     Created pod: svclb-shkeeper-external-2774e482-zqrzs
kube-system    42m         Normal    SuccessfulCreate       daemonset/svclb-traefik-8dbd47f2               Created pod: svclb-traefik-8dbd47f2-r5hqr
kube-system    42m         Normal    Scheduled              pod/svclb-shkeeper-external-2774e482-zqrzs     Successfully assigned kube-system/svclb-shkeeper-external-2774e482-zqrzs to shkeeper.anycolo.net
kube-system    42m         Warning   Evicted                pod/svclb-shkeeper-external-2774e482-zqrzs     The node had condition: [DiskPressure].
kube-system    42m         Warning   Evicted                pod/svclb-traefik-8dbd47f2-r5hqr               The node had condition: [DiskPressure].
kube-system    42m         Normal    Scheduled              pod/svclb-traefik-8dbd47f2-r5hqr               Successfully assigned kube-system/svclb-traefik-8dbd47f2-r5hqr to shkeeper.anycolo.net
kube-system    27m         Warning   FailedDaemonPod        daemonset/svclb-shkeeper-external-2774e482     Found failed daemon pod kube-system/svclb-shkeeper-external-2774e482-zqrzs on node shkeeper.anycolo.net, will try to kill it
kube-system    27m         Warning   FailedDaemonPod        daemonset/svclb-traefik-8dbd47f2               Found failed daemon pod kube-system/svclb-traefik-8dbd47f2-r5hqr on node shkeeper.anycolo.net, will try to kill it
kube-system    27m         Normal    SuccessfulDelete       daemonset/svclb-shkeeper-external-2774e482     Deleted pod: svclb-shkeeper-external-2774e482-zqrzs
kube-system    27m         Normal    SuccessfulDelete       daemonset/svclb-traefik-8dbd47f2               Deleted pod: svclb-traefik-8dbd47f2-r5hqr
kube-system    27m         Normal    SuccessfulCreate       daemonset/svclb-shkeeper-external-2774e482     Created pod: svclb-shkeeper-external-2774e482-2ppbj
kube-system    27m         Normal    SuccessfulCreate       daemonset/svclb-traefik-8dbd47f2               Created pod: svclb-traefik-8dbd47f2-hsjk9
kube-system    27m         Normal    Scheduled              pod/svclb-shkeeper-external-2774e482-2ppbj     Successfully assigned kube-system/svclb-shkeeper-external-2774e482-2ppbj to shkeeper.anycolo.net
kube-system    27m         Warning   Evicted                pod/svclb-shkeeper-external-2774e482-2ppbj     The node had condition: [DiskPressure].
kube-system    27m         Normal    Scheduled              pod/svclb-traefik-8dbd47f2-hsjk9               Successfully assigned kube-system/svclb-traefik-8dbd47f2-hsjk9 to shkeeper.anycolo.net
kube-system    27m         Warning   Evicted                pod/svclb-traefik-8dbd47f2-hsjk9               The node had condition: [DiskPressure].
cert-manager   26m         Warning   FailedScheduling       pod/cert-manager-cainjector-59855d5697-b2bfb   0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
cert-manager   26m         Warning   FailedScheduling       pod/cert-manager-776d69949d-5z7cd              0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
shkeeper       25m         Warning   FailedScheduling       pod/shkeeper-deployment-76c995f6bd-hb5pw       0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
shkeeper       25m         Warning   FailedScheduling       pod/ethereum-shkeeper-86997b49dc-rw7jw         0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
shkeeper       27m         Warning   FailedScheduling       pod/ethereum-6d79dd666b-mjjsz                  0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
cert-manager   26m         Warning   FailedScheduling       pod/cert-manager-webhook-6d5454d55d-4cltx      0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
shkeeper       24m         Warning   FailedScheduling       pod/mariadb-84cc69cc4c-7fmrk                   0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
shkeeper       25m         Warning   FailedScheduling       pod/bnb-shkeeper-7d4ccbd5c4-j4w7b              0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
shkeeper       25m         Warning   FailedScheduling       pod/tron-shkeeper-5d5b778c74-v7h5w             0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
kube-system    12m         Warning   FailedDaemonPod        daemonset/svclb-shkeeper-external-2774e482     Found failed daemon pod kube-system/svclb-shkeeper-external-2774e482-2ppbj on node shkeeper.anycolo.net, will try to kill it
kube-system    12m         Warning   FailedDaemonPod        daemonset/svclb-traefik-8dbd47f2               Found failed daemon pod kube-system/svclb-traefik-8dbd47f2-hsjk9 on node shkeeper.anycolo.net, will try to kill it
kube-system    12m         Normal    SuccessfulDelete       daemonset/svclb-shkeeper-external-2774e482     Deleted pod: svclb-shkeeper-external-2774e482-2ppbj
kube-system    12m         Normal    SuccessfulDelete       daemonset/svclb-traefik-8dbd47f2               Deleted pod: svclb-traefik-8dbd47f2-hsjk9
kube-system    12m         Normal    SuccessfulCreate       daemonset/svclb-traefik-8dbd47f2               Created pod: svclb-traefik-8dbd47f2-ntpj6
kube-system    12m         Normal    Scheduled              pod/svclb-shkeeper-external-2774e482-j5pbt     Successfully assigned kube-system/svclb-shkeeper-external-2774e482-j5pbt to shkeeper.anycolo.net
kube-system    12m         Normal    SuccessfulCreate       daemonset/svclb-shkeeper-external-2774e482     Created pod: svclb-shkeeper-external-2774e482-j5pbt
kube-system    12m         Normal    Scheduled              pod/svclb-traefik-8dbd47f2-ntpj6               Successfully assigned kube-system/svclb-traefik-8dbd47f2-ntpj6 to shkeeper.anycolo.net
kube-system    12m         Warning   Evicted                pod/svclb-shkeeper-external-2774e482-j5pbt     The node had condition: [DiskPressure].
kube-system    12m         Warning   Evicted                pod/svclb-traefik-8dbd47f2-ntpj6               The node had condition: [DiskPressure].
default        3m46s       Warning   EvictionThresholdMet   node/shkeeper.anycolo.net                      Attempting to reclaim ephemeral-storage
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/monero-wallet-rpc        waiting for first consumer to be created before binding
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/bor-config-volume        waiting for first consumer to be created before binding
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/heimdall-config-volume   waiting for first consumer to be created before binding
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/lightning-lnd            waiting for first consumer to be created before binding
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/monerod                  waiting for first consumer to be created before binding
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/avalanchego-volume       waiting for first consumer to be created before binding
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/bnb-datadir              waiting for first consumer to be created before binding
shkeeper       3m39s       Normal    WaitForFirstConsumer   persistentvolumeclaim/lightning-rtl            waiting for first consumer to be created before binding
shkeeper       39s         Normal    ScalingReplicaSet      deployment/shkeeper-deployment                 Scaled down replica set shkeeper-deployment-76c995f6bd to 0 from 1
shkeeper       39s         Normal    ScalingReplicaSet      deployment/tron-shkeeper                       Scaled up replica set tron-shkeeper-95b77596b to 1
shkeeper       38s         Warning   FailedScheduling       pod/shkeeper-deployment-76c995f6bd-hb5pw       skip schedule deleting pod: shkeeper/shkeeper-deployment-76c995f6bd-hb5pw
shkeeper       39s         Normal    SuccessfulCreate       replicaset/tron-shkeeper-95b77596b             Created pod: tron-shkeeper-95b77596b-7d6df
shkeeper       38s         Warning   FailedScheduling       pod/tron-shkeeper-95b77596b-7d6df              0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
shkeeper       39s         Normal    SuccessfulDelete       replicaset/shkeeper-deployment-76c995f6bd      Deleted pod: shkeeper-deployment-76c995f6bd-hb5pw
shkeeper       39s         Normal    ScalingReplicaSet      deployment/shkeeper-deployment                 Scaled up replica set shkeeper-deployment-67964f864c to 1
shkeeper       39s         Normal    SuccessfulCreate       replicaset/shkeeper-deployment-67964f864c      Created pod: shkeeper-deployment-67964f864c-xqkxh
shkeeper       38s         Warning   FailedScheduling       pod/shkeeper-deployment-67964f864c-xqkxh       0/1 nodes are available: 1 node(s) had untolerated taint {node.kubernetes.io/disk-pressure: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..

[anycolo]

I think i actually ran out of space. I'm switching to connecting to a remote eth node. How do i delete the old data of the ethereum node?