Added Resnet50 closed division github action #289
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
|
Hi @sujik18 , were you able to run the GitHub action for resnet50 closed division successfully? It could be tried in your forked repo also. I think the dataset size would be a concern here for GitHub runner due to limited storage. |
|
Yes, the only option is to do this on a self hosted system. Alternatively we can go for a short closed submission covering all the scenarios. |
|
@sujik18 , as arjun mentioned, could you test this with short run? |
|
Hi @anandhu-eng, I would try running the test with short run on another branch. After fixing the issue, I will push the changes. |
| with: | ||
| fetch-depth: 0 | ||
|
|
||
| - name: Check if the submission is for closed division |
There was a problem hiding this comment.
What is this step doing? This is not a submission repo right?
There was a problem hiding this comment.
Yes, you are right. Earlier, I had the wrong impression that we needed to evaluate this action only for submissions in the closed division. However, as you correctly mentioned, this is not a submission repository, and we require this action to verify the correctness of the automation script. I have addressed this issue in the latest commit.
| - name: Test MLPerf Inference ResNet50 (Linux/macOS) | ||
| if: matrix.os != 'windows-latest' | ||
| run: | | ||
| mlcr run-mlperf,inference,_submission,_short,_all-scenarios, division=closed, --submitter="MLCommons" --pull_changes=yes --pull_inference_changes=yes --hw_name="gh_${{ matrix.os }} x86" --model=resnet50 --implementation=${{ matrix.implementation }} --backend=${{ matrix.backend }} --device=cpu --scenario=Offline --test_query_count=500 --target_qps=1 -v --quiet |
There was a problem hiding this comment.
Is this command tested?
There was a problem hiding this comment.
Apologies for opening the PR without testing the GitHub action earlier. I wasn't completely aware of how the GitHub action workflow operated, which led to this oversight. Here are the test logs for the latest commit: Logs
Fix for #247