Skip to content

feat(executor): add query execution stats #18345

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 14 commits into
base: main
Choose a base branch
from

Conversation

dqhl76
Copy link
Collaborator

@dqhl76 dqhl76 commented Jul 11, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

feat(executor): add query execution stats

root@localhost:8000/default/default> from system.query_execution;

from
  system.query_execution

╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│          node          │          timestamp         │               query_id               │ process_rows │ process_time_in_micros │
│         String         │          Timestamp         │                String                │    UInt64    │         UInt64         │
├────────────────────────┼────────────────────────────┼──────────────────────────────────────┼──────────────┼────────────────────────┤
│ vU7wCmWh9Zo1Vq2iGZbyN1 │ 2025-07-15 07:42:32.000000 │ bc8ddfc1-75c1-458d-8104-d4cd97b6bc69 │            2 │                    766 │
│ vU7wCmWh9Zo1Vq2iGZbyN1 │ 2025-07-15 07:42:38.000000 │ 77e2ad37-ae73-4a1d-a2b0-4cfdce4cceec │      9791888 │                 383508 │
│ vU7wCmWh9Zo1Vq2iGZbyN1 │ 2025-07-15 07:42:39.000000 │ 77e2ad37-ae73-4a1d-a2b0-4cfdce4cceec │    291175190 │                9891921 │
│ vU7wCmWh9Zo1Vq2iGZbyN1 │ 2025-07-15 07:42:40.000000 │ 79909746-d352-477a-8891-cc12a88d520e │            8 │                    448 │
│ vU7wCmWh9Zo1Vq2iGZbyN1 │ 2025-07-15 07:42:40.000000 │ 77e2ad37-ae73-4a1d-a2b0-4cfdce4cceec │    290045495 │                9887434 │
│ vU7wCmWh9Zo1Vq2iGZbyN1 │ 2025-07-15 07:42:41.000000 │ 77e2ad37-ae73-4a1d-a2b0-4cfdce4cceec │    291243909 │                9888362 │
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Jul 11, 2025
@dqhl76 dqhl76 force-pushed the refactor-executor-2 branch 7 times, most recently from d8a9c27 to 15ac573 Compare July 14, 2025 06:15
@dqhl76 dqhl76 added the ci-benchmark Benchmark: run all test label Jul 14, 2025
Copy link
Contributor

Docker Image for PR

  • tag: pr-18345-e6916cc-1752477930

note: this image tag is only available for internal use.

@dqhl76 dqhl76 force-pushed the refactor-executor-2 branch from 15ac573 to 5696744 Compare July 15, 2025 06:59
}
}
let worker_id = context.get_worker_id();
global.push_tasks(worker_id, Some(current_tasks), next_tasks);
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: This commit includes a minor fix unrelated to the PR's title.

Problem: Tasks were being incorrectly pushed to the current_tasks queue here. It could cause a single task cost two points to be executed.

Solution: This change removes the redundant push to current_tasks, ensuring each task is cost one point. (Only occurred at steal_task_to_context)

This reverts commit 5696744.
@dqhl76 dqhl76 removed the ci-benchmark Benchmark: run all test label Jul 15, 2025
@@ -110,42 +110,3 @@ if [ "$response2_data" != "[]" ] && [ "$response2_data" != "null" ]; then
else
echo "✓ response2 data field is empty as expected"
fi

# **External -> Internal**: should not reset
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove a flaky test: This test need wait for acquiring all semaphares. It can correct acquire finally but the time is not sure.

@dqhl76 dqhl76 marked this pull request as ready for review July 15, 2025 07:51
@dqhl76 dqhl76 requested a review from zhang2014 July 15, 2025 07:51
@dqhl76 dqhl76 force-pushed the refactor-executor-2 branch from 0176290 to 2e6f3a1 Compare July 15, 2025 09:36
@wubx
Copy link
Member

wubx commented Jul 16, 2025

Why not add it to system.processes?

@dqhl76
Copy link
Collaborator Author

dqhl76 commented Jul 16, 2025

Why not add it to system.processes?

The key difference is that system.processes shows a real-time snapshot of what's currently running, whereas system.query_execution displays performance metrics from the last 10 seconds. A finish query will also have a record.
Another reason is that in a distributed environment, system.processes only shows one record for a given query. The new table is designed to show a record for each node involved, giving a much more detailed, per-node view.
To not break the current system.processes, I choose to create a new table.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants