Increase report interval of spaming logs to 10 seconds #51012
+4
−4
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
In this PR I want to increase the default report interval of both
K8S
andApache Yarn
from
1s
to10s
.Why are the changes needed?
Due to the logs of type:
Being printed for every 1 seconds, the amount of logs made our
Airflow
's UI slow,it also takes too much space as we save the spark logs for future use.
I think the report interval should be increased to make a better balance between notifying the user
that the application is running and not spamming us.
I think it should be done globally to prevent other users to go what we went through:
If we would update the default, we would ease the use for many users in my opinion.
Does this PR introduce any user-facing change?
Yes, in this PR we increase the report interval from 1 second to 10 seconds.
The log of type:
Would be printed every 10 seconds and not every 1 second by default.
But the users can change it if they want to, and they shouldn't be affected by it as it's just logging.
How was this patch tested?
I have tesed the patch manually,
I have an
Airflow
Cluster with docker, a k8s cluster. With that, I have created a spark submit connection,and created a DAG that uses the spark submit operator
to run spark on my k8s.
The important thing that I have done is that I have added the config:
To my spark application at the spark submit operator conf.
Which made the report interval to increase to 10 seconds from 1 second.
Was this patch authored or co-authored using generative AI tooling?
No.
This is my first PR, if there's a problem, please notify me.