Reuse dvc.yaml with differnt params.yaml files per scenario

[SebBanDev]

Hello I have a project where I want to use the same dvc.yaml with different params.yaml files

In my dvc.yaml I use different input files based on the path of the scenario in the deps:
data/${schenario_folder}/....csv
and also some other params as arg parse commands in some stages:

    cmd: >-
      python project/ml/prepare_data.py
      ${start_date}
      ${timespan}

I want to run the same stages for different params.yaml files with dvc repro.

So far I had a copy of the same dvc.yaml in every folder:

├───dvc
│   ├───scenario_1
│   │       dvc.lock
│   │       dvc.yaml
│   │       params.yaml
│   │
│   └───scenario_2
│           dvc.lock
│           dvc.yaml
│           params.yaml

using e.g. for scenario_1 dvc repro dvc\scenario_1\dvc.yaml

Isn't there a way to use the same dvc.yaml by specifiying the path ot the params.yaml (and where to output dvc.lock) so that I could use:

├───dvc
|   |   dvc.yaml
|   |
│   ├───scenario_1
│   │       dvc.lock
│   │       params.yaml
│   │
│   └───scenario_2
│           dvc.lock 
│           params.yaml

using e.g. for scenario_1 dvc repro dvc\dvc.yaml --params dvc\scenario_1\params.yaml --lock dvc\scenario_1\dvc.lock

*cross post on discord, but I got no answer, so I thought I might try it over here

[isidentical]

Isn't there a way to use the same dvc.yaml by specifiying the path ot the params.yaml (and where to output dvc.lock) so that I could use:

As far as I know, we don't support using delagated dvc.lock files to elsewhere (and multiple of them). You can load params.yaml, like this at the top of your pipeline

vars:
  - params.json
  - config/myapp.yaml

but from what I understand, you have multiple values for the same param so that might not be the best fit. From what I can understand by your description, I think foreach stages might help you: https://dvc.org/doc/user-guide/project-structure/pipelines-files#foreach-stages

Where you can have a single params.yaml, and you can make each param a list and go over them (e.g instead of source_dir: source/ on params1.yaml, and source_dir: source2/, you can just make source_dirs: ['source/', 'source2/'] and go over it with foreach``).

[SebBanDev]

Thanks for the idea, but it's not quite what we want to do, we still need to be able to run just one of the scenarios. But for now, just committing the same dvc.yaml in different folders with different params.yaml respectively works fine as workaround.

[dberenbaum]

By the way, you can run one scenario at a time with foreach by doing dvc repro mystage@scenario_1. You would still have a single dvc.lock file at the same level as dvc.yaml, but it would version each scenario as a separate stage.

[adamelen]

@SebbanSms Had exactly the same question with you; any chance you have answered it by now?

[SebBanDev]

We still use the setup where we have different scenarios sorted in different folders, each having their own
dvc.yaml, dvc.lock and params.yaml whereas the params change and the dvc.yaml is always the same.

We stopped working on that project. The only workaround I can think of is storing the dvc.yaml file one level up and using a script that moves it to the scenario folder, executes pipeline, moves it one level up afterwards again. But I guess that is pretty hacky.

[adamelen]

Ok thank you! :)

[OlofHarrysson]

I'm also looking to achieve a similar scenario. I'm looking to run my pipeline in normal mode as well as have the option to run it in "fast-mode".

The pipeline is built in several stages and I would like to have the option to only run the fast-branch pipeline (in all stages).

Any ideas?