Skip to content

[SW-230999] mark_step is required for acc and perf issue #1453

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Jun 18, 2025
Merged

[SW-230999] mark_step is required for acc and perf issue #1453

merged 1 commit into from
Jun 18, 2025

Conversation

leopck
Copy link

@leopck leopck commented Jun 18, 2025 •
edited by github-actions bot
Loading

QA is observing 10% acc drop on Llava v1.5 7B as well as 2hr time taken for execution from 3mins previously.

Jira: https://jira.habana-labs.com/browse/SW-230999

Root cause is due to requiring mark_step to break the graph, actual cause is still under investigation but this fixes the issue.

================= 1 passed, 904 warnings in 283.15s (0:04:43) ==================

Results

{"task_name": "mmmu_val", "accuracy": 0.332, "target_accuracy": 0.331, "performance": 51, "valid": true, "batch_size": "1"}

@leopck
Copy link
Author

leopck commented Jun 18, 2025

/run-gaudi-tests

@leopck
mark_step is required for Llava v1.5 7B
7ad234e 
QA is observing 10% acc drop on Llava v1.5 7B as well as 2hr time taken
for execution from 3mins previously.

Root cause is due to requiring mark_step to break the graph, actual
cause is still under investigation but this fixes the issue.

Signed-off-by: Chendi Xue <chendi.xue@intel.com>
Signed-off-by: Stanley Phoong <stanley.cheong.kwan.phoong@intel.com>
@leopck
Copy link
Author

leopck commented Jun 18, 2025

/run-gaudi-tests

1 similar comment
@xuechendi
Copy link

/run-gaudi-tests

@xuechendi xuechendi enabled auto-merge (squash) June 18, 2025 19:25
@xuechendi xuechendi merged commit 527c1ee into habana_main Jun 18, 2025
52 checks passed
@xuechendi xuechendi deleted the markstepllava branch June 18, 2025 21:15
@leopck leopck changed the title [Llava v1.5 7B] mark_step is required for acc and perf issue [SW-230999] mark_step is required for acc and perf issue Jun 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@xuechendi xuechendi xuechendi approved these changes

@kzawora-intel kzawora-intel Awaiting requested review from kzawora-intel kzawora-intel is a code owner

@madamczyk-intel madamczyk-intel Awaiting requested review from madamczyk-intel madamczyk-intel is a code owner

@michalkuligowski michalkuligowski Awaiting requested review from michalkuligowski michalkuligowski is a code owner

@mgawarkiewicz-intel mgawarkiewicz-intel Awaiting requested review from mgawarkiewicz-intel mgawarkiewicz-intel is a code owner

@vivekgoe vivekgoe Awaiting requested review from vivekgoe vivekgoe is a code owner

@afierka-intel afierka-intel Awaiting requested review from afierka-intel afierka-intel is a code owner

@jikunshang jikunshang Awaiting requested review from jikunshang jikunshang is a code owner

@mswiniarsk mswiniarsk Awaiting requested review from mswiniarsk mswiniarsk is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@leopck @xuechendi