A validator becomes very slow to handle a new transaction right after rejoining and syncing the network

[byron1st]

Discord user ID

byron1st

Describe your question in detail.

Hello, I'm running a private blockchain network using 8 Aptos validators.
I found that a validator becomes very slow or unable to handle a transaction right after rejoining and syncing the network.

I found this situation following the below steps:

1. Remove the pod of a validator (I used k8s). This means, its data are also removed.
2. Then relaunch the pod of a validator. Keys and identities are same, so it started to a state sync process.
3. During the state syncing, this validator cannot handle an incoming transaction.
4. After its syncing is done (I figure out if the state sync is completed or not by calling the GET /v1 API and checking the epoch and version), it seems normal for this new validator. However, when I sent a transaction to this validator, it is extremely slow to handle the transaction, or unable to handle it.

When extremely slow to handle a transaction, the logs of this validator seems normal, but sometimes it produces warning logs to indicate that out bound RPC call fails.
When unabling to handle it, the validator produced warning logs to indicate that this validator is a sync_only status.

This situation is resolved when a new epoch starts. So, I doubt that the network holds the participation of a newly syncing validator until the next epoch. Is it correct?

Thanks.!

What error, if any, are you getting?

No response

What have you tried or looked at? Or how can we reproduce the error?

No response

Which operating system are you using?

macOS

Which SDK or tool are you using? (if any)

Aptos CLI

Describe your environment or tooling in detail

No response

[gregnazario]

A new validator can only join at a new epoch to the validator set. Cc @JoshLind for some more detail here

[byron1st]

Hi, @gregnazario !
Thanks for your reply.
The validator that I synced is not a totally new validator, but an existing validator that just sync from the genesis again. Its keys and identities are not changed, so the network will recognize this validator as an existing one.