Skip to content

Increase report interval of spaming logs to 10 seconds #51012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Avihais12344
Copy link

What changes were proposed in this pull request?

In this PR I want to increase the default report interval of both K8S and Apache Yarn
from 1s to 10s.

Why are the changes needed?

Due to the logs of type:

Application status for (phase: )

Being printed for every 1 seconds, the amount of logs made our Airflow's UI slow,
it also takes too much space as we save the spark logs for future use.
I think the report interval should be increased to make a better balance between notifying the user
that the application is running and not spamming us.
I think it should be done globally to prevent other users to go what we went through:

  1. Getting this bug.
  2. Finding if there's a way to reduce the logs.
  3. Finding this config (I went streight to the source code, only to find it's someplace at the docs).
  4. Updating it in our applications.

If we would update the default, we would ease the use for many users in my opinion.

Does this PR introduce any user-facing change?

Yes, in this PR we increase the report interval from 1 second to 10 seconds.
The log of type:

Application status for (phase: )

Would be printed every 10 seconds and not every 1 second by default.
But the users can change it if they want to, and they shouldn't be affected by it as it's just logging.

How was this patch tested?

I have tesed the patch manually,
I have an Airflow Cluster with docker, a k8s cluster. With that, I have created a spark submit connection,
and created a DAG that uses the spark submit operator
to run spark on my k8s.
The important thing that I have done is that I have added the config:

{
"spark.kubernetes.report.interval": "10s"
}

To my spark application at the spark submit operator conf.
Which made the report interval to increase to 10 seconds from 1 second.

Was this patch authored or co-authored using generative AI tooling?

No.

This is my first PR, if there's a problem, please notify me.

@Avihais12344
Copy link
Author

Avihais12344 commented May 25, 2025

I have enabled the actions in my fork, but I can't rerun the failed job, please help.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant