Downstream tasks should be set instead of list

[jay0129]

I am new to airflow.
I have a suggestion about the syntax of the way we define downstream tasks.

task_a >> [task_b, task_c]
When I am looking at the above code, I thought like after execution of task_a, there will be task_b triggered and then task_c, which is misunderstanding.

So how about the following code ?
task_a >> {task_b, task_c}

Plus, Tree View of a DAG in the Airflow web console shows non sequential order as well:

[potiuk]

Nope. Dependency of tasks is denoted by >>, << so what you described would be:

task_a >> task_b >> task_c

The set vs. List makes no difference there, I think you should be able to use both list and set, what matters is that this is iterable, of a number of tasks.

[BasPH]

In addition to @potiuk's answer, the behaviour you're referring to can be achieved with airflow.models.baseoperator.chain():

task1 = DummyOperator(task_id="task1")
task2 = DummyOperator(task_id="task2")
task3 = DummyOperator(task_id="task3")
chain(task1, task2, task3)
# equal to task1 >> task2 >> task3

Which is often used in combination with a for loop:

tasks = [DummyOperator(task_id=f"task{i}") for i in range(10)]
chain(*tasks)

[jay0129]

To make my question clear.
What I suggested is task_a >> {task_b, task_c} looks more suitable than task_a >> [task_b, task_c] as the order of task_b and task_c doesn't important.
This is because list has index but set didn't.

[potiuk]

Yep. I perfectly realized that intention..

The answer is that it does not make any difference for us since we chose a bit different semantics (which is not pythonic anyway) with >> rather than sequence in list denoting dependencies. And lists and sets are for us just iterables of tasks where sequence does not mean dependency (and anything else).

This is de-facto a new small domain language of dependencies. And in this language we as community (and it was looooong before i joined) we consciously departed from basing on sequentiality of array vs .set and the dependencies are denoted not by array sequence but by >> << operators.

I believe even now you could use both lists and sets if you want and we make no difference (but I might be wrong on that).

If we follow your example (and line of thought) we should be able to write [ a, {b, c} ] and get the same dependencies.

The problem with lists that you need really an operator (such as >>) that set dependencies. If you just put tasks in an array there is no way you can build such dependency (you need to call a method on the operator instance to add dependency to the other operator). So we really need to use << >> for dependencies.

So there is really no semantic meaning to list elements sequence in our case (and can't be due to the way python treats lists) - so to avoid even thinking that list can be treated as sequence of tasks, we removed that notion in our domain language - lists are just iterables of tasks, so it is perfectly fine to use them to build such dependencies to multiple tasks. If we start distinguishing between lists and sets, that might create a confusion and questions 'why can't i do [ a, {b, c}] ?'

[potiuk]

Speaking of which. I just double-checked and it got me interested.

Maybe we COULD change it to accept both set and lists -> we actually useSequence for that operator now, so Set would not work actually. But If we change it to Iterable, it could work. I wonder what others think here - should we also allow 'set` in this ? I think there is no point in having two tasks repeated in same list anyway, so Set will do as well as List here, and we seem to only use Iterable part of the Sequence with those operators.

Maybe other long timers, could shed some light on it ? Is there any reason we only iimit the << operator to "Sequeneces" ? I looked historically and it seems that we historically only used lists, and then we changed it to Sequences (@turbaszek) but maybe going further to Iterables would make sense as well?

[BasPH]

Lol was trying the exact same :-) I don't see any reason not to use sets for grouping operators too, makes perfect sense to me.

[potiuk]

@jay0129 -> maybe you can create a feature (or even try to open PR straight away with support for sets? It should be simple change of the type supported + some documentation). Sounds like nice contribution!