Allow Matrix to be defined within a step itself

[Andre601]

Idea

It could be a useful feature to allow defining a Matrix within a step itself, instead of only for the whole workflow itself.

One main reason is to reduce duplicate actions that could easily be done within a single step alone, rather than re-doing them within each matrix itself.

The syntax would effectively be the same as with the global setting, just within a step itself, meaning that a matrix setup like this:

matrix:
  include:
    - FOO: lorem
      BAR: ipsum
    - FOO: dolor
      BAR: sit
    - FOO: amet
      BAR: consetetur

...would look like this when included within a step:

steps:
  matrixStep:
    image: example-image
    matrix:
      include:
      - FOO: lorem
        BAR: ipsum
      - FOO: dolor
        BAR: sit
      - FOO: amet
        BAR: consetetur
    commands:
      - echo "$FOO $BAR"

Problems

There would obviously be some problems that may or may not be avoidable.
I don't have the technical knowledge to say for certain if the mentioned issues are valid or not.

Matrix values could only be used within commands/steps?

With these matrix options being defined within a step itself, it would obviously be limited to just this step alone and couldn't be used across the entire file like with the global one.

Global and per-step matrix mixing

This could be an issue where a user may want to try and use a matrix for the file globally, while also wanting to have one within a step.
This setup could result in n1*n2 individual steps where n1 is the global matrix number and n2 the step matrix number (So a setup using 3 entries on global matrix and 4 entries on step matrix would result in 12 runs, only assuming a single step with matrix configuration is used).

I believe that such combinations should be treated as an invalid file configuration by Woodpecker to avoid possible abuse cases where a person creates a global matrix with n entries and multiple steps with n steps...
In general should it perhaps be considered to limit number of matrix configurations within a step to 1... mainly because of...

Problems defining matrix step -> matrix step progression

...how should a situation like this be handled?

steps:
  matrixStep:
    image: example-image
    matrix:
      include:
      - FOO: lorem
        BAR: ipsum
      - FOO: dolor
        BAR: sit
      - FOO: amet
        BAR: consetetur
    commands:
      - echo "$FOO $BAR"
  anotherMatrixStep:
    image: example-image
    matrix:
      include:
      - FOO: sadipscing
        BAR: elitr
      - FOO: sed
        BAR: diam
      - FOO: nonumy
        BAR: eirmod
    commands:
      - echo "$FOO $BAR"

If we would just make a matrix step for each previous matrix step would this be the result:

graph LR
    lorem["lorem + ipsum"]
    dolor["dolor + sit"]
    amet["amet + consetetur"]

    sad1["sadipscing + alitr"]
    sad2["sadipscing + alitr"]
    sad3["sadipscing + alitr"]
    sed1["sed + diam"]
    sed2["sed + diam"]
    sed3["sed + diam"]
    nonumy1["nonumy + eirmod"]
    nonumy2["nonumy + eirmod"]
    nonumy3["nonumy + eirmod"]

    echo1>"echo "lorem ipsum""]
    echo2>"echo "lorem ipsum""]
    echo3>"echo "lorem ipsum""]
    echo4>"echo "dolor sit""]
    echo5>"echo "dolor sit""]
    echo6>"echo "dolor sit""]
    echo7>"echo "amet consetetur""]
    echo8>"echo "amet consetetur""]
    echo9>"echo "amet consetetur""]

    echo10>"echo &quotMsadipscing alitr""]
    echo11>"echo &quotMsadipscing alitr""]
    echo12>"echo &quotMsadipscing alitr""]
    echo13>"echo "sed diam""]
    echo14>"echo "sed diam""]
    echo15>"echo "sed diam""]
    echo16>"echo "nonumy eirmod""]
    echo17>"echo "nonumy eirmod""]
    echo18>"echo "nonumy eirmod""]

    lorem --> echo1 --> sad1 --> echo10
    lorem --> echo2 --> sed1 --> echo13
    lorem --> echo3 --> nonumy1 --> echo16

    dolor --> echo4 --> sad2 --> echo11
    dolor --> echo5 --> sed2 --> echo14
    dolor --> echo6 --> nonumy2 --> echo17

    amet --> echo7 --> sad3 --> echo12
    amet --> echo8 --> sed3 --> echo15
    amet --> echo9 --> nonumy3 --> echo18

Loading

3 separate steps produced by the first step, which then results in three steps for each caused by the next step.
Depending on what is done in the commands could this significantly increase stress on the CI, which is why I repeat my suggestion of limiting matrix usage within a step itself to one step only.

How should files propagate?

This is another difficult decision here... If the matrix involves creating and managing files, how should they propagate for non-matrix steps?
Should the follow-up steps be duplicated for each matrix entry? Should all matrix merge back?

Perhaps this could be regulated by a setting that decides how the matrix step should complete here, with default being duplicating the follow-up steps and run in parallel.

To give a visual example again, using this setup here:

steps:
  makeFiles:
    image: example-image
    matrix:
      include:
      - FOO: lorem
        BAR: ipsum
      - FOO: dolor
        BAR: sit
      - FOO: amet
        BAR: consetetur
    commands:
      - echo "$FOO > $BAR.txt"
  readFile:
    image: example-image
    commands:
      - './readfiles.sh' # Assume this would find any TXT file and print their content to console.

would give this flow if follow-up steps are duplicated for each matrix entry:

graph LR
    lorem["lorem + ipsum"]
    dolor["dolor + sit"]
    amet["amet + consetetur"]

    echo1>"echo "lorem > ipsum""]
    echo2>"echo "dolor > sit""]
    echo3>"echo "amet > consetetur""]

    print1>"./readfile.sh"]
    print2>"./readfile.sh"]
    print3>"./readfile.sh"]

    result1(["lorem"])
    result2(["dolor"])
    result3(["amet"])

    lorem --> echo1 --> print1 --> result1
    dolor --> echo2 --> print2 --> result2
    amet --> echo3 --> print3 --> result3

Loading

Meanwhile with merging the matrixes back together in 1 follow-up step to use, this would be the flow:

graph LR
    lorem["lorem + ipsum"]
    dolor["dolor + sit"]
    amet["amet + consetetur"]

    echo1>"echo "lorem > ipsum""]
    echo2>"echo "dolor > sit""]
    echo3>"echo "amet > consetetur""]

    print>"./readfile.sh"]

    result1(["lorem"])
    result2(["dolor"])
    result3(["amet"])

    lorem --> echo1 --> print --> result1
    dolor --> echo2 --> print --> result2
    amet --> echo3 --> print --> result3

Loading

The question with merging would be, what to do with conflicting files?
Perhaps a config option to define this could be used with the option being to fail the run, but also allowing overrides or "merging" them (i.e. appending values together).

Order would be the same as with the list, meaning that with above example (And assuming the same file is used for all matrix entries) and using override, lorem would go first, them dolor would override it followed by amet overriding dolor.

How to handle failed runs in a matrix?

Probably the same as with current global matrix settings... Maybe also allow to define the action: cancel run, ignore (continue), ...?

Actual example use-case

I'm using a possible use-case here I myself would've had in the past (Tho not anymore now) to give an idea of why such a feature could be beneficial.
I cannot guarantee this example is valid in terms of parameters and alike...

when:
  - event: tag
    branch: master

steps:
  createFiles:
    image: maven:3.9.4-eclipse-temurin-17
    commands:
      - mvn clean install # Build the files
  uploadToSites:
    image: leplusorg/json
    matrix:
      include: # Not sure if and/or how you could obtain secrets in matrix...
        - URL: example.com/api/v1/upload/
          TOKEN: some_secret_token
          FILE_DIRECTORY: example
        - URL: another.example.com/api/v2/new-file/
          TOKEN: another_secret_token
          FILE_DIRECTORY: another_example
        - URL: yet-another.example.com/api/v3/uploading/
          TOKEN: yetanother_secret_token
          FILE_DIRECTORY: yet_another_example
    commands:
      - |-
        curl -X POST \
        --header "Authorization: $TOKEN" \
        --header "Content-Type: multipart/form-data" \
        --form "attachment="@$FILE_DIRECTORY/to/a/file.exe" -s -S \
        --fail-with-body \
        "https://$URL"

Closing words

I hope the examples and descriptions I gave make it clear what my idea is. I want to also reiterate that I'm aware of possible issues and limitations that would need to be thought about here... in fact did some issues pop up in my mind while I was writing some of the examples...

I understand if this would be a too complex or too niche thing to consider. Was just some "food for thought" I would say...

The idea is inspired by GitHub Actions' support of matrix per-step, which works since it doesn't have the "files are taken to the next step" situation here... which is also a downside of sorts.