Support experiment enablement in Taskfile schema

[nathanperkins]

Description

Experiments like #1038 cause major changes to the behavior for a task. Given that the taskfile would have to be written to make use of this experiment, it is dangerous that the experiment has to be enabled using env vars for every engineer and every automation flow which runs that task. If the task were accidentally run without the experiment, it could lead to undesired behavior, like deploying remote environments with settings based on a developer's local environment.

The risk of dangerous, undesired behavior makes it practically impossible to use the experiment, so we have to continue using a workaround.

Ideally, we would be able to enable this experiment within the taskfile itself so that we have consistent behavior across our team and automation without any risk.

[pd93]

(Was just about to reply in #1038), but may as well have this discussion here now.

This is why we have option 3 in our docs for enabling experiments. This allows you to commit the .env file which enables it for anyone checking out your repo. I do understand that a .env file isn't always ideal though.

My main concern with adding them to the Taskfile schema itself is that it would (ironically) be a backwards-compatible change to remove the experiments later which we need to be able to do.

Maybe a compromise could be to add support for a dedicated .task-experiments.yaml file or something similar.

[nathanperkins]

This is why we have option 3 in our docs for enabling experiments. This allows you to commit the .env file which enables it for anyone checking out your repo. I do understand that a .env file isn't always ideal though.

Ah, I see. I thought the .env file would only be used if we sourced it ourselves or if we included it in dotenv but I didn't want to replace any other vars.

Based on my experimentation, it seems like the experiment vars are loaded from .env regardless of whether you put it in dotenv, while other variables are ignored unless you have it in dotenv. It might be worth clarifying this further in the docs.

If so, that's good enough for my team.

My main concern with adding them to the Taskfile schema itself is that it would (ironically) be a backwards-compatible change to remove the experiments later which we need to be able to do.

That makes sense but also maybe this kind of change should break visibly. I'd rather task exit with an error than for it to do something unexpected.

Maybe a compromise could be to add support for a dedicated .task-experiments.yaml file or something similar.

That sounds good, although I'm fine with the current .env solution as I understand it.

[nathanperkins]

Possibly related: #1979

[vmaerten]

My main concern with adding them to the Taskfile schema itself is that it would (ironically) be a backwards-compatible change to remove the experiments later which we need to be able to do.

Adding that handling remote experiment directly in the Taskfile schema could be tricky.

I wonder if it would be better to have a more global YAML configuration file, like .task-config.yml or .task-settings.yaml. This could help address issues like this one #1589
Any thoughts @pd93 / @andreynering ?

[pd93]

Seems like the general consensus is that we do not want to support this at a schema-level. It is already possible to do this using config files and #1982 will extend this further. There is also #2014 to track enabling using the package API.