Add e2e test for train API #2199

helenxie-bit · 2024-08-09T03:12:53Z

What this PR does / why we need it:
Add an e2e test in the test_e2e_train_api.py for the train API.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

coveralls · 2024-08-09T03:34:29Z

Pull Request Test Coverage Report for Build 12449362407

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals
Change from base Build 12381609583:	0.0%
Covered Lines:	85
Relevant Lines:	85

💛 - Coveralls

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

helenxie-bit · 2024-12-19T03:43:03Z

Run the e2e test for the train API as a separate GitHub Action.

Can you try to separate them and see if issue will be resolved ? cc @kubeflow/wg-training-leads

@andreyvelich I've separated the e2e test for train API and now it works. Please review when you have time.

andreyvelich

Thank you for this overall lgtm, just small comment.
/assign @deepanker13 @kubeflow/wg-training-leads @Electronic-Waste

.github/workflows/e2e-test-train-api.yaml

deepanker13 · 2024-12-19T17:30:28Z

/lgtm
Thanks @helenxie-bit

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

helenxie-bit · 2024-12-20T04:45:56Z

I've updated the Kubernetes version to v1.31.4. Please review when you have time @andreyvelich @Electronic-Waste @kubeflow/wg-training-leads

Electronic-Waste

Basically LGTM. I left some comments for you @helenxie-bit

Electronic-Waste · 2024-12-20T06:56:49Z

.github/workflows/e2e-test-train-api.yaml

+    strategy:
+      fail-fast: false
+      matrix:
+        kubernetes-version: ["v1.31.4"]


Shall we change the Kubernetes version to be aligned with other ci tests? Like:

https://github.com/kubeflow/training-operator/blob/69094e16309382d929606f8c5ce9a9d8c00308b1/.github/workflows/test-example-notebooks.yaml#L16-L18

Just to save compute resources, I think for now we can just run this test on a single k8s version, since we run the rests E2E tests on the all versions.
WDYT @Electronic-Waste ?

Yeah, I agree. Maybe we can select one k8s version from this list:)

OK, let me change the version to v1.30.6. And we can update it if needed in the future.

given that we support 1.28-1.31, I would suggest that we run our integration tests on 1.29, 1.30, 1.31, we can update it in the following PR.
For the train API tests, I think running it on 1.31 should be sufficient.

Sounds good. So I think we will still keep the v1.31.4 version.

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

andreyvelich

Thank you for this effort @helenxie-bit!
/lgtm
/approve
/hold cancel

google-oss-prow · 2024-12-21T23:41:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andreyvelich,tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* add e2e test for train API Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix peft import error Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update settings of the job Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix error detection Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * resolve conflict Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * resolve conflict Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * resolve conflict Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix NoneType error Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * test bug Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * find bug Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * find bug Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * find bug Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * add storage_config Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * reduce pvc size Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * set storage_config Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * set storage_config Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * set storage_config Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * set storage_config Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * use gpu Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * use gpu Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * use gpu Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix 'set_device' error Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * add timeout error Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix typo Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update e2e test for train api Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * add num_labels Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update pip install Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check disk space Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * change sequence of e2e tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * add clean-up after each e2e test of pytorchjob Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function-add check disk Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check docker volumes Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check docker directory Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update pip install and 'num_workers' Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update pip install and 'num_workers' Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update pip install Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * change the value of 'clean_pod_policy' Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * change the value of 'update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check docker volumes Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check docker volumes Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * stop the controller and restart it again to clean up Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update cleanup function Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * separate e2e test for train api Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix parameter of namespace Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * reduce resources Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * separate e2e test for train API Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * remove go setup Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * adjust the version of k8s Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * move test file to new place Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix typos Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * rerun tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update install packages Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * build and verify images of storage-intializer and trainer Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix image build error Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix image build error Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check disk space Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * make 'setup-storage-initializer-and-trainer' executable Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * separate step of loading images Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check disk space after loading image Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * clean up and check disk space Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * prune docker build cache Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * prune docker build cache Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * adjust sequence of building and loading images Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * move working directory Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * delete moving working directory Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * use 'docker system prune' Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * make the format of the commands to be consistent Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update base image Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update base image Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update base image Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * delete unnecessary space clear and check code Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * merge e2e test for train api into integration tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * check for timeout error Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix name of trainer image Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix env of building storage initializer image Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * clean format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * skip e2e test for train API when use scheduling Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * Update name of fileholder Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com> Signed-off-by: Hezhi (Helen) Xie <hezxie@ucdavis.edu> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * separate e2e test for train API Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * fix format Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * move test script Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update path to test script Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update path to test script Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * rerun tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * rerun tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * rerun tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update kubernetes version Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * update kubernetes version Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * rerun tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * rerun tests Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * adjust kubernetes version to 1.30.6 Signed-off-by: helenxie-bit <helenxiehz@gmail.com> * adjust kubernetes version to 1.31.4 Signed-off-by: helenxie-bit <helenxiehz@gmail.com> --------- Signed-off-by: helenxie-bit <helenxiehz@gmail.com> Signed-off-by: Hezhi (Helen) Xie <hezxie@ucdavis.edu> Co-authored-by: Andrey Velichkevich <andrey.velichkevich@gmail.com>

add e2e test for train API

15b6cb0

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

google-oss-prow bot requested review from jinchihe and kuizhiqing August 9, 2024 03:13

google-oss-prow bot added the size/L label Aug 9, 2024

fix peft import error

daa0054

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

helenxie-bit added 3 commits August 9, 2024 12:05

update settings of the job

8d4af90

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

fix format

86c31c8

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

fix format

01870e2

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

helenxie-bit changed the title ~~Add e2e test for train API~~ [WIP]Add e2e test for train API Aug 9, 2024

google-oss-prow bot added the do-not-merge/work-in-progress label Aug 9, 2024

helenxie-bit changed the title ~~[WIP]Add e2e test for train API~~ [WIP] Add e2e test for train API Aug 9, 2024

helenxie-bit added 18 commits August 9, 2024 20:46

fix error detection

17f3c33

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

resolve conflict

0685dc7

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

resolve conflict

83de64b

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

resolve conflict

f954f2d

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

fix format

ff48154

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

fix NoneType error

304db5d

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

fix format

486154d

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

test bug

016c41d

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

find bug

1e7bd23

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

find bug

1aced61

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

find bug

3100aae

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

add storage_config

e5b9061

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

fix format

ffb0685

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

reduce pvc size

dc1b48a

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

set storage_config

8894517

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

set storage_config

36872d7

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

set storage_config

7dd8d40

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

set storage_config

60c322d

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

helenxie-bit added 4 commits December 18, 2024 18:55

update path to test script

617dba3

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

rerun tests

b5ae618

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

rerun tests

9b997a3

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

rerun tests

6e8f3f7

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

andreyvelich reviewed Dec 19, 2024

View reviewed changes

.github/workflows/e2e-test-train-api.yaml Outdated Show resolved Hide resolved

google-oss-prow bot assigned deepanker13 Dec 19, 2024

google-oss-prow bot added the lgtm label Dec 19, 2024

andreyvelich mentioned this pull request Dec 19, 2024

Add MNIST example with SPMD for JAX #2346

Closed

update kubernetes version

f4bb238

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

google-oss-prow bot removed the lgtm label Dec 20, 2024

helenxie-bit added 3 commits December 19, 2024 18:52

update kubernetes version

775ff67

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

rerun tests

fc21273

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

rerun tests

4c51b76

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

Electronic-Waste reviewed Dec 20, 2024

View reviewed changes

helenxie-bit added 2 commits December 21, 2024 15:36

adjust kubernetes version to 1.30.6

52c73b4

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

adjust kubernetes version to 1.31.4

13a0de2

Signed-off-by: helenxie-bit <helenxiehz@gmail.com>

andreyvelich reviewed Dec 21, 2024

View reviewed changes

google-oss-prow bot removed the do-not-merge/hold label Dec 21, 2024

google-oss-prow bot assigned andreyvelich Dec 21, 2024

google-oss-prow bot added the lgtm label Dec 21, 2024

google-oss-prow bot merged commit d7f69e8 into kubeflow:master Dec 22, 2024
55 checks passed

sandipanpanda mentioned this pull request Dec 27, 2024

Track follow-up for adding CI image build to the Kind cluster #2364

Closed

1 task

Add e2e test for train API #2199

Add e2e test for train API #2199

Uh oh!

Conversation

helenxie-bit commented Aug 9, 2024

Uh oh!

coveralls commented Aug 9, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 12449362407

Details

💛 - Coveralls

Uh oh!

helenxie-bit commented Dec 19, 2024

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

deepanker13 commented Dec 19, 2024

Uh oh!

helenxie-bit commented Dec 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Electronic-Waste left a comment

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Dec 20, 2024

Choose a reason for hiding this comment

Uh oh!

andreyvelich Dec 20, 2024

Choose a reason for hiding this comment

Uh oh!

Electronic-Waste Dec 21, 2024

Choose a reason for hiding this comment

Uh oh!

helenxie-bit Dec 21, 2024

Choose a reason for hiding this comment

Uh oh!

andreyvelich Dec 21, 2024

Choose a reason for hiding this comment

Uh oh!

helenxie-bit Dec 21, 2024

Choose a reason for hiding this comment

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

google-oss-prow bot commented Dec 21, 2024

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

coveralls commented Aug 9, 2024 •

edited

Loading

helenxie-bit commented Dec 20, 2024 •

edited

Loading