support dynamically pipeline ？

[Rentu]

The type of transform is fixed. Suppose there are 10 transforms, and these 10 transforms can form a pipeline, but the order might vary. If I have 10 different pipelines, I would need to write 100 transforms, which is not very flexible and might even affect performance. Is there a way to dynamically assemble the pipeline with these 10 transforms according to different sequences on a regular basis?

[pront]

I cannot think of a built-in feature that can help with this. It sounds like you need a layer that generates your final config.

Note that you can use * to match components, see combine_logs below:

sources:
  demo_logs_1:
    type: demo_logs
    format: json

  demo_logs_2:
    type: demo_logs
    format: json

transforms:
  combine_logs:
    type: remap
    inputs:
    - demo*
    source: |
      # remap

sinks:
  console:
    type: console
    inputs: ["combine_logs"]
    encoding:
      codec: json
      json:
        pretty: true