Add held to possible TaskInstanceState

[cdabella]

Description

Add held as a TaskInstanceState which functions similarly to failed but represents a stop in DAG execution that is known and planned.

Use case/motivation

Many DAGs and pipelines have steps that require human intervention and sign-off before continuing, like manual data validation or manager approval before continuation. This can be functionally achieved today by marking a task as failed, but operationally overloading the meaning of failed can cause issues with Ops/monitoring/alerting that may not have the complete picture to know whether a task has truly failed or has been marked failed by design. Adding an additional TaskInstanceState which represents putting a task on-hold improves clarity in DAG design while achieving the functional goal of stopping a DAGRun from continuing.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

This is not a feature to be implemented easily - this is really a discussion idea. There are a lot of things to consider here- it's not only "state" but a lot of other things - UI, API, consequences.

I personally believe this is a good idea to have but rather than merely a "feature", this calls for an entirely new Airflow Improvement Proposal https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals - that should describe the design, discuss edge cases, explain trade-offs and consequences and eventually seek consensus and voting on the Devlist as it ads entirely new flow to Airflow.

BTW. You can - even now do it via adding a Sensor waitng for something (file on S3, or FTP. Database row value, etc. ). With Deferrable Operators you can even implement it efficiently using Triggerer and Async.io without spending resources on active IO. But then you would have to implement your own notifications and UI for the user to ttigger such manual events.

So the scope of this "AIP" is really not about adding state but about all the UI and workflow changes to make such workflow (maybe) posiible to be managed vi Airflow UI.

Maybe others will chime in here, but if you are really into it @cdabella - making a more detailed proposal /document with pros/cons/consequences, proposed UI changes etc. and sharing it on a devlist is a good thing. Or maybe someone else who will see it and pick an interest in it will do it.

[cdabella]

Thanks for the feedback @potiuk, and apologies for the misrepresentation. It's definitely more of an Epic as opposed to an individual feature. I'm still new to the AIP process, but I'll take a look.

I've considered the sensor model, which in some respects makes sense (hah) as you are waiting on external state, specifically human action. IMHO, it doesn't capture an operators goal of managing DAGs without structurally modifying those DAGs. I've seen many instances now where the DAG developers are not the ones actually operating the DAGs day-to-day usage, so being able to separate those two duties becomes important in the design.

Thanks again for the feedback, I'll take a look at an AIP.

[potiuk]

No need to apologise (but It's a good sign that you read what I wrote and get to know the process :) . This is what "convert to discussion" Github Issue feature is great for. We do not expect everyone to know our process when they start interacting with the project, that's why I also explained in detail what expectations generally are for this kind of change.

I've considered the sensor model, which in some respects makes sense (hah) as you are waiting on external state, specifically human action. IMHO, it doesn't capture an operators goal of managing DAGs without structurally modifying those DAGs. I've seen many instances now where the DAG developers are not the ones actually operating the DAGs day-to-day usage, so being able to separate those two duties becomes important in the design.

Yes. I fully agree that "authoring" and "operating" DAGs are two diffrent roles more often than not (which is even reflected in Airflow's permissions and security model and will be even more in the future). And both roles are different than "deploying" Airflow as well. All three roles in any serious installation of Airlfow will be done by different people/teams. We are well aware of that.

And I would say even more - providing a "feature" for DAG Authors that they could just use to make "human-in-the-loop" interaction possible with just giving the "operations" people easy to use feture without unnecessary involving "deployment" people to add extra "deployment component" - should be one of the most important motivations on why the sensor-based approach is not enough and why Airflow UI-based solution (and integration) should be better..

And I think this should be major reason in the AIP why we want to implement it. This is a very good point you made and I think when/if you prepare AIP this should be front and center in the "why" part.

BTW. IMHO the easiest (and sustainable/cool) solution, that should leverage what we have now is actually IS to use sensor rather than introduce a new state. Adding state is problematic - there are multiple queries and places in Airflow Internals that assume the set of existing States. Adding new state means that you have to adapt code in many, many places and add a lot of tests to cover that . Sensor is much better, but in order to fulfill the easines of passing it between roles, it would be great to have specialized sensor that does it, which is automatically handled by the UI to show it in the right way to the "operations" people (without the need of "deployments" people doing anything. But I might not know everything and it might have some other problems, so ths is precisely why AIP and devlist discussions are needed - not everyone follows Github discussion /issues and this is an important feature that goes cross whole airlfow stack.