Ad-hoc CWL workflows not applicable

[fmigneault]

Using CWL for describing workflows allows more flexibility that then Part 3 Nested Processes approach (#445).

Part 3 mentions a "Requirement for CWL ad-hoc workflow definition". However, that makes no sense, since "ad-hoc" implies submitting the execution values directly, whereas CWL is a Workflow Definition, similarly to Process Description. CWL are not convenient with injected values. They instead rely on a separate "execution/job" document, similar to the OAP's Execution Request.

How to provide both of these documents simultaneously needs to be addressed to make CWL definitions actually usable.

More details here:

• Support ad-hoc CWL workflows crim-ca/weaver#834 (comment)

Comment with Possible Workarounds:

• Support ad-hoc CWL workflows crim-ca/weaver#834 (comment)

[jerstlouis]

@fmigneault So I believe we are both clear on the use case of defining a workflow using CWL and then deploying that as a process, and then exexuting this using Part 1 or Part 3 "ad-hoc workflows" to reference this new process (and possibly other processes), right?

The use case that was envisaged for this "CWL ad-hoc workflow definition" is to define a CWL workflow referencing processes already deployed, with fixed/literal execution values.

Are you saying that CWL does not have the flexibility to hardcode process inputs and/or to reference processes already deployed?

[fmigneault]

Yes, deploying the workflow definition first has no issue, since values will be provided after during execution. Once deployed, the OAP interface generated from the CWL can be provided, so that process can be involved within other OAP ad-hoc workflows.

CWL allows use valueFrom to hardcode some inputs, but this requires the user to not forget adding the required StepInputExpressionRequirement to support it.

You can reference other CWL within workflow steps using run: <ref>, but in pure CWL, <ref> would have to be a local CWL file. In our implementation, we supercharge that to allow any URI, including a OGC API - Processes ProcessDescription endpoint, though the conversion must be handled explicitly by the server since no CWL-native tool will understand it.

[jerstlouis]

@fmigneault

You can reference other CWL within workflow steps using run: , but in pure CWL, would have to be a local CWL file. In our implementation, we supercharge that to allow any URI, including a OGC API - Processes ProcessDescription endpoint, though the conversion must be handled explicitly by the server since no CWL-native tool will understand it.

Well that would be exactly the idea of what we would specify in this Requirement class "CWL ad-hoc workflow definition".

We would be explicitly extending CWL to support such URIs, and with this valueFrom for specifying input, it seems that this could all actually work in practice?

[fmigneault]

It would work partially for simple cases (i.e.: literal values).

For type: File inputs (the equivalent of OAP href), it could be tricky. Normally, submitting the OAP href allows the server to do necessary steps in between the request and the CWL tool's invocation to fetch the file and stage it locally. Using valueFrom sets the path string directly. If the underlying tool does not handle remote file retrieval, the resulting invocation will not work.

The CWL approach would also be limited for collection inputs since there no similar concept in it. Normally, a process definition able to receive a single collection URI has to be mapped (internally) to a CWL array of files (type: File[]) to allow the workflow to execute the process steps in parallel. This makes the use of valueFrom rather unhelpful if using the CWL directly, since you have to set valueFrom using a complex JavaScript generating the array from a list of pre-resolved URIs that would otherwise be obtained from the collection. Since the whole point of using collection is to let that resolution happen dynamically, it is not practical to force pre-resolving the references by the user. The collection URI could not be used directly as valueFrom, since that causes an ambiguity with the previous type: File variant (href).

[jerstlouis]

@fmigneault My thinking is that this needs some prototyping / code sprinting to see how it "could" work for those more complicated href and collection inputs, but that in theory, it should be possible to come up with requirements and guidance detailing how this should be implemented. So I would suggest that we leave this issue open and keep thinking about it until the next code sprint (which I hope will focus intensely on Part 3!).

a single collection URI has to be mapped (internally) to a CWL array of files (type: File[]) to allow the workflow to execute the process steps in parallel.

That needs to happen before passing this to the CWL workflow runner, but presumably the OGC API - Processes - Part 3 implementation would have another layer which fetches data from the input collections, write this to temporary internal files, and generates internal CWL which converts this to the type: File[] to give it to the CWL runner.

The collection URI could not be used directly as valueFrom, since that causes an ambiguity with the previous type: File variant (href).

We could surely disambiguate this somehow? Either with a qualifier or some new OGC API - Processes specific construct?

In my opinion, this ad-hoc CWL workflow req. class makes a lot of sense in theory, but yes it extends beyond typical CWL usage. However, we have no current plans ourselves to implement anything to do with CWL. So if you do not see this as ever being useful and no one else does either, we can consider removing it or moving it to future work.

[fmigneault]

That needs to happen before passing this to the CWL workflow runner, but presumably the OGC API - Processes - Part 3 implementation would have another layer which fetches data from the input collections, write this to temporary internal files, and generates internal CWL which converts this to the type: File[] to give it to the CWL runner.

There's no problem there if they happen at separate steps. The issue is that, if the CWL type: File[] is what you submit directly as ad-hoc workflow, you cannot simultaneously provide valueFrom: <collection-uri> and the array type. They are not formed the same way (an array of strings vs just the collection URI string). The other workflow operations that rely on that input array will be malformed CWL if valueFrom: <collection-uri> was used. Therefore, you need to have both type: File[] and <collection-uri> provided on their own and separately to allow this intermediate step to happen.

We could surely disambiguate this somehow? Either with a qualifier or some new OGC API - Processes specific construct?

This would require defining a CWL CommandInputRecordSchema to represent the collection. Since submitting CWL, you would have to deal with their approach of defining custom properties rather than OGC API - Processes specific constructs.

In the meantime, I'm gradually considering the "Request Variant using Multipart" and "Request Variant using Embedded Process" approaches detailed in crim-ca/weaver#834 (comment). Those would offer the most flexibility since CWL can be submitted normally (as if pre-deploying it, without valueFrom), and allows OGC-specific utilities like collection input to be leveraged simultaneously.

The valueFrom is sometimes used in pre-deployed CWL for certain use cases. I'm not a big fan of overloading it for the specific use of ad-hoc, nor all the edge cases it seems to introduce.

So if you do not see this as ever being useful and no one else does either, we can consider removing it or moving it to future work.

On the contrary, I am raising this issue because I have an actual requirement arising for some projects that would leverage this approach, but needs some clarifications where the definitions are still rough around the edges.

[jerstlouis]

On the contrary, I am raising this issue because I have an actual requirement arising for some projects that would leverage this approach, but needs some clarifications where the definitions are still rough around the edges.

Perhaps we could retitle this issue more positively then?

"Clarify how Ad-hoc CWL workflows could be applicable" or similar :)

[fmigneault]

I would consider it a clarification if it was only a matter of ambiguity, but I find the current definition is not applicable as it stands because there is more than just grey areas. Just semantics though. Evaluations at the level of code sprints would be needed to validate.