Please add user custom return value for TrainerClient().train run

[ram4444]

What you would like to be added?

I have just gone through some sample codes of TrainerClient v2. However, I cant find any example of returning some useful metric like loss/accuracy of the distributed training run. Currently only the job id is returned

job_id = TrainerClient(namespace="kubeflow-system").train

or any better suggestions?

Why is this needed?

to pass the metrics back to the katib objective

Love this feature?

Give it a 👍 We prioritize the features with most 👍

[andreyvelich]

Thank you for your interest in KF Trainer @ram4444!
Currently, TrainerClient doesn't allow you to save metrics. If you want to use it with Katib, you can report metrics two ways:

• Print metrics and use pull-based metrics collector: https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/#pull-based-metrics-collector
• Using push-based metrics collector: https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/#push-based-metrics-collector

Additionally, you can use tools like W&B to report metrics from your Experiments: https://docs.wandb.ai/guides/track/

[andreyvelich]

/remove-label lifecycle/needs-triage
/area sdk

[ram4444]

Thank you for your interest in KF Trainer @ram4444! Currently, TrainerClient doesn't allow you to save metrics. If you want to use it with Katib, you can report metrics two ways:

• Print metrics and use pull-based metrics collector: https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/#pull-based-metrics-collector
• Using push-based metrics collector: https://www.kubeflow.org/docs/components/katib/user-guides/metrics-collector/#push-based-metrics-collector

Additionally, you can use tools like W&B to report metrics from your Experiments: https://docs.wandb.ai/guides/track/

Hi,

Thanks for your quick reply. This really helps.

https://www.kubeflow.org/docs/components/katib/getting-started/
When I first look at the Getting Start code, I think there is something automatically collect the "result" variable specified at the function call but never think of the line print(f"result={result}") is for doing so. I think it is for debug purpose only 😆

I will never comment out any line of print statement in machine learning code in the future 😂

[andreyvelich]

That's great to hear @ram4444!
Do you want to contribute support of TrainJob in Katib Experiments ?
Currently, we don't support it out-of-the-box: https://www.kubeflow.org/docs/components/katib/user-guides/trial-template/

[ram4444]

Do you mean creating a pull request to add the following to katib controller rbac?

• apiGroups:
  - trainer.kubeflow.org
  resources:
  - trainjobs
  verbs:
  - "get"
  - "list"
  - "watch"
  - "create"
  - "delete"

And also katib/manifests/v1beta1/installs/katib-standalone/katib-config.yaml?

trialResources:
TrainJob.v2alpha1.trainer.kubeflow.org

(Why i cannot find it in kubeflow manifest apps/katib?)

[andreyvelich]

That's right @ram4444!
I think, we should support TrainJob CRD (TrainJob.v1alpha1.trainer.kubeflow.org) by default.
Adding some simple example, also would be super useful!

(Why i cannot find it in kubeflow manifest apps/katib?)

We haven't integrated TrainJob CRD into Katib yet.

[andreyvelich]

cc @franciscojavierarceo @kubeflow/kubeflow-trainer-team @astefanutti

[ram4444]

Katib PR
KF manifest PR

I hope my code submission comes closed to error free
Please let me know if any correction/enhancement can be done.