Skip to content

feat: Add perf task to generate perf data. #1134

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat: Add perf task to generate perf data. #1134

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
ceb02cd
add synced taskfile
PacificViking Jul 28, 2025
1cf1913
add task for perf (building with symbols), bump yscope-dev-utils vers…
PacificViking Jul 28, 2025
0ccd943
fix yaml linting errors
PacificViking Jul 28, 2025
655517a
fix according to taskrabbit
PacificViking Jul 30, 2025
b35f771
remove stamp during cleaning
PacificViking Jul 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 53 additions & 1 deletion taskfile.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ vars:
G_CORE_COMPONENT_SUBMODULES_DIR: "{{.G_CORE_COMPONENT_DIR}}/submodules"
G_WEBUI_SRC_DIR: "{{.G_COMPONENTS_DIR}}/webui"

G_DATASET_LOCATION: "{{.ROOT_DIR}}/datasets/dataset.log"

# Build paths
G_BUILD_DIR: "{{.ROOT_DIR}}/build"
G_CORE_COMPONENT_BUILD_DIR: "{{.G_BUILD_DIR}}/core"
Expand Down Expand Up @@ -50,6 +52,7 @@ tasks:
clean:
cmds:
- "rm -rf '{{.G_BUILD_DIR}}'"
- "rm -f .task/build_with_symbols_*.stamp"
- task: "clean-python-component"
vars:
COMPONENT: "clp-package-utils"
Expand Down Expand Up @@ -94,6 +97,7 @@ tasks:
vars:
CHECKSUM_FILE: "{{.G_BUILD_DIR}}/{{.TASK}}.md5"
OUTPUT_DIR: "{{.G_PACKAGE_BUILD_DIR}}"
BUILD_WITH_SYMBOLS: "{{.BUILD_WITH_SYMBOLS | default false}}"
sources:
- "{{.G_BUILD_DIR}}/package-venv.md5"
- "{{.G_BUILD_DIR}}/webui.md5"
Expand All @@ -111,7 +115,9 @@ tasks:
- "components/package-template/src/**/*"
generates: ["{{.CHECKSUM_FILE}}"]
deps:
- "core"
- task: "core"
vars:
BUILD_WITH_SYMBOLS: "{{.BUILD_WITH_SYMBOLS}}"
- "clp-package-utils"
- "clp-py-utils"
- "init"
Expand Down Expand Up @@ -204,11 +210,17 @@ tasks:
ref: ".OUTPUT_DIRS"

core:
vars:
BUILD_WITH_SYMBOLS: "{{.BUILD_WITH_SYMBOLS | default false}}"
cmds:
- task: "core-generate"
vars:
BUILD_WITH_SYMBOLS: "{{.BUILD_WITH_SYMBOLS}}"
- task: "core-build"

core-generate:
vars:
BUILD_WITH_SYMBOLS: "{{.BUILD_WITH_SYMBOLS | default false}}"
internal: true
sources: &core_source_files
- "{{.G_DEPS_CORE_CHECKSUM_FILE}}"
Expand All @@ -217,14 +229,23 @@ tasks:
- "{{.G_CORE_COMPONENT_DIR}}/src/**/*"
- "{{.TASKFILE}}"
- "/etc/os-release"
generates:
# ugly workaround so that when the var changes, the task gets re-run (triggering core-build
# re-run)
- ".task/build_with_symbols_{{.BUILD_WITH_SYMBOLS}}.stamp"
deps:
- "clp-s-generate-parsers"
- "deps:core"
cmds:
- task: "utils:cmake:generate"
vars:
EXTRA_ARGS:
- >-
{{if eq .BUILD_WITH_SYMBOLS "true"}}-DCMAKE_BUILD_TYPE=RelWithDebInfo{{end}}
BUILD_DIR: "{{.G_CORE_COMPONENT_BUILD_DIR}}"
SOURCE_DIR: "{{.G_CORE_COMPONENT_DIR}}"
- "rm -f .task/build_with_symbols_*.stamp"
- "touch .task/build_with_symbols_{{.BUILD_WITH_SYMBOLS}}.stamp"

core-build:
internal: true
Expand Down Expand Up @@ -586,6 +607,37 @@ tasks:
cmds:
- "rm -rf dist"

download-dataset:
status:
- "test -f {{.G_DATASET_LOCATION}}"
cmds:
- >-
curl --fail --location --show-error --silent
--output output.tar.gz
https://zenodo.org/records/10516387/files/cockroachdb.tar.gz?download=1
- "tar -xzf output.tar.gz"
- "rm -f output.tar.gz"
- >-
mkdir -p "$(dirname "{{.G_DATASET_LOCATION}}")"
- "mv cockroachdb/cockroach.node1.log {{.G_DATASET_LOCATION}}"
- "rmdir cockroachdb"

Comment on lines +610 to +624
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

rmdir cockroachdb will error ‑- tarball contains multiple files

The CockroachDB archive ships several .log files.
After moving only cockroach.node1.log, the directory is not empty, so rmdir returns exit status 1 and the task fails.

-      - "mv cockroachdb/cockroach.node1.log {{.G_DATASET_LOCATION}}"
-      - "rmdir cockroachdb"
+      - "mv cockroachdb/cockroach.node1.log {{.G_DATASET_LOCATION}}"
+      - "rm -rf cockroachdb"

Optional but recommended:

  • add a checksum to detect a corrupted download
  • quote {{.G_DATASET_LOCATION}} in the test -f status check for path safety
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
download-dataset:
status:
- "test -f {{.G_DATASET_LOCATION}}"
cmds:
- >-
curl --fail --location --show-error --silent
--output output.tar.gz
https://zenodo.org/records/10516387/files/cockroachdb.tar.gz?download=1
- "tar -xzf output.tar.gz"
- "rm -f output.tar.gz"
- >-
mkdir -p "$(dirname "{{.G_DATASET_LOCATION}}")"
- "mv cockroachdb/cockroach.node1.log {{.G_DATASET_LOCATION}}"
- "rmdir cockroachdb"
download-dataset:
status:
- "test -f {{.G_DATASET_LOCATION}}"
cmds:
- >-
curl --fail --location --show-error --silent
--output output.tar.gz
https://zenodo.org/records/10516387/files/cockroachdb.tar.gz?download=1
- "tar -xzf output.tar.gz"
- "rm -f output.tar.gz"
- >-
mkdir -p "$(dirname \"{{.G_DATASET_LOCATION}}\")"
- "mv cockroachdb/cockroach.node1.log {{.G_DATASET_LOCATION}}"
- "rm -rf cockroachdb"
🤖 Prompt for AI Agents 
In taskfile.yaml lines 610 to 624, the command `rmdir cockroachdb` fails because
the tarball contains multiple files and the directory is not empty after moving
only one log file. Replace `rmdir cockroachdb` with a command that forcefully
removes the entire directory and its contents, such as `rm -rf cockroachdb`.
Additionally, quote `{{.G_DATASET_LOCATION}}` in the `test -f` status check to
handle paths safely, and optionally add a checksum verification step after
downloading to ensure file integrity.

perf:
deps:
- "download-dataset"
cmds:
- task: "package"
vars:
BUILD_WITH_SYMBOLS: true
- "rm -f perf.data"
- >-
perf record -F 99 -g -- ./build/core/clp-s c --timestamp-key "timestamp"
--target-encoded-size 268435456 build/clp_cockroachdb_perf_output "{{.G_DATASET_LOCATION}}"
- "chmod 644 perf.data"
- >-
perf report -i perf.data --call-graph=graph,0.5,0,caller,function,percent --stdio --demangle
--no-inline > perf-report.txt

init:
internal: true
silent: true
Expand Down
2 changes: 1 addition & 1 deletion tools/yscope-dev-utils
Open in desktop
Submodule yscope-dev-utils updated 29 files
+2 −2 .github/ISSUE_TEMPLATE/bug-report.yaml
+50 −0 .github/workflows/code-linting-checks.yaml
+5 −0 .github/workflows/unit-tests.yaml
+35 −2 README.md
+9 −0 exports/lint-configs/python/README.md
+8 −0 exports/lint-configs/python/mypy.ini
+27 −0 exports/lint-configs/python/ruff.toml
+76 −72 exports/taskfiles/utils/boost.yaml
+12 −7 exports/taskfiles/utils/cmake.yaml
+20 −76 exports/taskfiles/utils/cpp-lint.yaml
+7 −5 exports/taskfiles/utils/remote.yaml
+6 −6 exports/taskfiles/utils/utils.yaml
+1 −0 exports/ystdlib-py/.gitignore
+1 −0 exports/ystdlib-py/.python-version
+83 −0 exports/ystdlib-py/README.md
+57 −0 exports/ystdlib-py/pyproject.toml
+1 −0 exports/ystdlib-py/src/ystdlib/__init__.py
+140 −0 exports/ystdlib-py/src/ystdlib/pyfind.py
+94 −0 exports/ystdlib-py/uv.lock
+1 −0 lint-requirements.txt
+15 −0 taskfile.yaml
+99 −82 taskfiles/boost/test_boost.cpp
+53 −0 taskfiles/lint/main.yaml
+22 −0 taskfiles/lint/python.yaml
+22 −0 taskfiles/lint/yaml.yaml
+65 −12 taskfiles/remote/tests.yaml
+6 −0 taskfiles/tests.yaml
+143 −0 taskfiles/ystdlib-py/test-pyfind.yaml
+10 −0 taskfiles/ystdlib-py/tests.yaml