Support for Asynchronous Orchestration Event with a "Wait" Requisite

[grichmond-salt]

Description of Issue/Question

While asking a question about parallel tasks during an orchestration state, the following ideas were discussed between Tom, Seth, and Mike:

An async execute plus a wait_for_event would be a simple and killer Orchestrate addition. It could be implemented as a requisite or a state.

The wait requisite would be evaluated when the state gets evaluated, but a wait state would be evaluated when it gets evaluated, which COULD be before it should be evaluated so it could make an earlier lock. However, having a wait state allows us to be a lot more flexible in what we wait for and how we wait. It also allows us to define a single wait that many other states can require.

A requisite addition would also be awesome. The ability to extract data from the event and then do something with it (e.g., check success/fail/changes) would be very helpful.

Perhaps we should mandate that it would always be the last evaluated requisite. This makes sense, because you might not be able to wait for something that you also require. This is because you would need to require the thing that would eventually trigger the wait state.

It could be hard to chain the waits in a deterministic way without a deep understanding of requisite chain ordering, which is something we cannot expect people to understand. This, perhaps, is an argument in support of a wait requisite, but requisites are just not made to inherently have arguments and the conditions of the wait are critical to the orchestration.

We could also add a feature to do requisite ordering. With this in place, one could explicitly say: "evaluate this requisite last". Since the order of requisite eval is right now hard coded into the state system it is like state ordering used to be; deterministic but non-intuitive.

[mbakke]

I have a state that should be applied only when required by another state (which has appropriate unless or onchanges requisites).

The use case for me is temporarily overriding another state. In particular, temporarily change permissions on a managed folder for a cmd.run that runs as a restricted user.

It looks something like this:

dir1:
  file.directory:
    - name: /some/path
    - mode: 555
dir1_override:
  file.directory:
    - name: /some/path
    - mode: 755
    - require:
      - dir1
    - use:
      - dir1
    - wait: # only run when called from one of these requisites
      - cmd: some-cmd
some-cmd:
  cmd.run:
    - runas: nonroot
    - onchanges:
      - file: /some/other/managed/file
    - require:
      - file: dir1_override

Not sure if this is helpful, but it's a real use case that doesn't seem to have any good solutions currently. In this case wait could implicitly establish a require_in dependency and vice versa for the wait_in counterpart to aid with ordering.

[onlyanegg]

I've also been looking for something like this. I use the orchestrate runner for rolling application deployments. Some services can take several minutes to come up. I have a semi-working solution for this but it would be nice to have the minion send an event when the service is fully up. Then the orchestrate runner could go on to the next box.

[onlyanegg]

For my use case (application deployments of multiple services in multiple "pods" with orchestrate), I think it would easier and more clear to define the order in which states or functions are run rather than deal with requisites. I have to wait until the first application is deployed and then I can deploy the rest. I imagine it could look something like this:

deploy_app_master:
  salt.state:
    - tgt: app_master
    - sls:
      - deploy.app_master
    - order: 1

deploy_app:
  salt.state:
    - tgt: app*
    - sls: 
      - deploy.app
    - order: 2

[aphor]

Is anyone currently working on this?

[aphor]

I've been thinking about this a lot lately, because the only solutions using the current capabilities I can think of for rolling updates to a multi-tier application are

• python sls orchestration, with some crazy stuff in it, like injecting reactors at render time and then deleting them afterwards, which will be a PAIN to debug, and make robust.
• custom thorium module which compiles low chunks, and injects some additional requisite behavior around events, which also gets ugly on the whiteboard every time I try to approach it.

The problem is that the orchestration state runs on the master, and its requisites are scoped to the highstate run on the master, but we need some of those requisites to care about the results of states running remotely on potentially many minions. Events are a convenient way for states on minions to communicate something to the master, but state requisites are used on the master to compile and run orchestration low chunks, before anything can fire events.

I thought about adding a def block() to salt/state/event.py (because "wait" is taken by the mod_watch semantics), but if we want to be truly asynchronous (read: avoid weird race behavior), that will need to salt.utils.event.get_event(...) to start listening at the beginning of the orchestration low chunks, or at least before anything which might trigger the event, so we're back in the weeds either doing something new to reorder low chunks, or trying to get existing requisites code to perform the magic for us. This feels messy too.

What I feel like would be ideal, is wrapping low chunks in coroutines that return a {'not yet':{'awaiting':['an/event/to match'], 'passed':['an/event/already/matched']}} instead of a result, and polling them in a job mini-reactor loop iterating on the event listener, which would first send them the event, and then check their result. If the coroutine got an event it was 'awaiting' it would move that to the passed list, and if the awaiting list was empty, it would run the low chunk payload and return results.

The coroutines with event requisites would get created with 'awaiting' reservations, and low chunks without them would just return results. I don't think this will coexist well with the way low chunks are compiled. State requisites are not async now, and mixing in async blocking might be messy. I think maybe making state requisites event based might be a best first step. Maybe in an experimental state compiler just for orchestration at first? Bonus: it may be possible to parallelize state runs without multiprocessing..

Am I overthinking this? Comments?

[aphor]

I wrote a quick and dirty runner function as a PoC. I also provided a "hello world" orchestration roll.sls to demonstrate how this works with orchestration states.

https://gist.github.com/aphor/9e94a6aef663546575f5d0a13af1e825.js

Observations:

• This chokes on the requisite recursion of call_chunk() in the state compiler when I try to make all the states parallel: True (I suspect because of artifacts related to multiprocessing pickling and un pickling)
• Other weird stuff happens when I try to make just the async steps or the waitfor states parallel: True
• Each waitfor salt.function state needs to spin up its own event listener, and it would be ideal to start ONE per orchestration runner, adding subscriptions and get_event() calls one each per state.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[max-arnold]

Bump

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[Oloremo]

I think it's related to the #49273

[stale[bot]]

Thank you for updating this issue. It is no longer marked as stale.

[aphor]

I might be ready to restart work on this soon if there's any continued interest. Please let me know because I might need help testing if I can come up with a PR.

[Oloremo]

@aphor would be killer feature for us, ready to help with testing