Is there a way to log container defined in ResourceTemplate?

[mirocody]

Hi there,

Currently I am trying to building pipeline with argo and kubeflow operator. As part of the workflow, I defined the training job in resourcetemaplate.

  - metadata:
      labels:
        user: xxx
    name: step4
    resource:
      action: create
      failureCondition: failed in (status.replicaStatuses.Master, status.replicaStatuses.Worker)
      manifest: |
        apiVersion: kubeflow.org/v1
        kind: PyTorchJob
        metadata:
          generateName: step4-pytorchjob-
          labels:
            user: xxx
          namespace: xxx
        spec:
           <pytorchjob related spec>

It works, but in Argo UI, it can only log the container of that "argo container", which looks like below:

time="2021-10-21T07:01:06.195Z" level=info msg="0/1 success conditions matched"
time="2021-10-21T07:01:06.195Z" level=info msg="Waiting for resource pytorchjob.kubeflow.org/pytorchjob-l8kc9 in namespace argo resulted in retryable error: Neither success condition nor the failure condition has been matched. Retrying..."
time="2021-10-21T07:01:11.199Z" level=info msg="Get pytorchjobs 200"
time="2021-10-21T07:01:11.199Z" level=info msg="failure condition '{failed in [status.replicaStatuses.Master status.replicaStatuses.Worker]}' evaluated false"
time="2021-10-21T07:01:11.199Z" level=info msg="success condition '{status.replicaStatuses.Master.succeeded exists []}' evaluated false"
time="2021-10-21T07:01:11.199Z" level=info msg="0/1 success

However, it does not support the log from pytorchjob container. I am curious if I missed something, or argo does not support this feature for now? Thanks.

[terrytangyuan]

TFJob is a custom resource so I don't think logs are available besides the main container log that basically watches for the resource status. I'd suggest using a sidecar to collect your logs from worker/master pods and inject the log to a separate container/step so that you can see via the UI. Other than that, kubectl is the easiest way to access the logs.

The current solution for this is through a sidecar, e.g. example mentioned in #6910 (comment)

    - name: tfjob-multiworker-sidecar
      sidecars:
        - name: master
          image:  argoproj/argoexec:v3.0.7
          command:
            - sh
            - -c
          args:
            - "kubectl wait --for=condition=ContainersReady pod --selector job-name={{workflow.name}},job-role=master && kubectl logs -l job-name={{workflow.name}},job-role=master --tail=1000 -f"

Related discussions #6910 and #7012

[mirocody]

As this is a duplicate topic of #7012 ， I will mark this as answered and discuss more there.