Wait for events inside a spawned thread

[eduwercamacaro]

From what I can see, the current design for external events requires user to specify both nodeRunNumber and threadRunNumber when posting an external event for a specific WfRun. I believe this approach makes user experience harder because these values are difficult to infer. Consequently, wait for events inside a spawned thread is even more difficult because the threadRunNumber will change on every execution of a wfspec. For example:

SpawnedThreads spawnedThreads = wf.spawnThreadForEach(approvalChain.jsonPath("$.approvals"), "spawn-threads", innerThread -> {
                    innerThread.waitForEvent("my-inner-event");
                },
                Map.of("not-used-variable", 1234)
                );

Should we post multiple events on this thread? or a single external event should complete multiple nodeRuns?

I believe a single external event should complete any ExternalEventNodeRun when users don't specify a threadRunNumber or nodeRunNumber

[eduwercamacaro]

pinging @Snarr and/or @coltmcnealy-lh to review this topic.

[coltmcnealy-lh]

requires user to specify both nodeRunNumber and threadRunNumber when posting an external event for a specific WfRun

Close, but not quite. Both thread_run_number and node_run_position are optional. You can also specify thread_run_number without node_run_position. (if you specify node_run_position you must also specify thread_run_number.)

---

This discussion makes me think that we want to have the ability to do something like the following inside the workflow DSL:

    public void childThreadLogic(WorkflowThread wf) {
        LHFormatString callbackToken = wf.format("{0}-{1}", wf.getId(), wf.getThreadRunNumber());
        wf.execute("send-docusign", documentId, callbackToken);
        wf.waitForEvent("document-signed");
    }

In particular, the two additions are:

• WorkflowThread#getWfRunNumber()
• WorkflowThread#getWfRunId()

We have a really old ticket for the first one already: #581

[Snarr]

Hi Eduwer, I agree that in your example it seems very difficult to post an external event.

So your proposal is to complete any ExternalEventNodeRun when users don't specify a ThreadRunNumber or NodeRunNumber.

Could we expand this further and say that if users don't specify a NodeRunNumber, but they specify a ThreadRunNumber, then we will complete any ExternalEventNodeRun on that ThreadRun?

[eduwercamacaro]

Ohh that's an excellent observation. It makes total sense to me. We can do that.

[coltmcnealy-lh]

Weighing in here—the currently-documented behavior is exactly what @Snarr is suggesting (I know because I wanted it to work that way, and I remember writing it to work that way in the past, and the documentation says so). Using a specific node_run_position is exceedingly rare, and the only time you would want to do that is:

• a single ThreadRun waits for the same ExternalEventDef multiple times.
• Something happened which you know will only fix the "second" time you get to that event but not the "first"

Look at the protobuf documentation again:

message PutExternalEventRequest {
  // ...

  // Optionally specify that this ExternalEvent may only be claimed by a specific ThreadRun.
  optional int32 thread_run_number = 6;

  // Optionally specify that this ExternalEvent may only be claimed by a specific NodeRun.
  // In order for this to be set, you must also set thread_run_number.
  optional int32 node_run_position = 7;
}

It's possible that the behavior doesn't match the documentation since I don't think we have an e2e test for this. We probably should write one.

[coltmcnealy-lh]

About the original post: we should maintain the current default behavior where one ExternalEvent can only be matched to a single ExternalEventNode. We have users in production whose use-cases depend on that behavior (I cannot share these specific use-cases).

If you want, we can keep the current default behavior the same and allow users to opt-in to "posting one ExternalEvent will signal to all ExternalEventNodes waiting on that event."

Some discussion points, as this would require some really careful design. Note that I don't have opinions on these behaviors; I just think they need to be designed, documented, and tested carefully:

1: Where Behavior is Configured

There are three places we can conceivably configure this behavior, making no comment on which one makes the most sense:

1. At the ExternalEventNode level inside the ThreadSpec.
2. At the rpc PutExternalEvent level.
3. At the ExternalEventDef level.

Option 1 has the following complication:

• If threadSpec foo has the default behavior and threadSpec bar has the new behavior (one ExternalEvent triggers all nodes)
• If there's a thread of type foo and a few threads of type bar waiting on the event,
• and the event comes, then what happens?

Option 2 makes it risky because if a WfSpec needs the new behavior (one ExternalEvent triggering all waiting nodes) and a client uses the default behavior, the WfRun will never complete.

Option 3 is less intuitive but it doesn't have those dangers.

2: Race Between Threads and Events

The current behavior of External Events is that if an ExternalEvent comes in before a ThreadRun gets to the "ExternalEventNode" (which waits for that event), it will "pick up" the previously-posted ExternalEvent and continue.

How will that work in @eduwercamacaro 's proposal? In particular, with the following scenario:

• There are three child ThreadRuns (1, 2, and 3) that are going to wait for the document-signed event.
• Threads 1 and 2 get to that node.
• The event comes in.
• Then thread 3 gets to that node.

Should thread 3 complete, or should it be stuck?

3: Repeating the Same Event

If a ThreadSpec looks like this:

wf.execute("send-docusign");
wf.waitForEvent("document-signed");
wf.execute("send-another-docusign");
wf.waitForEvent("document-signed");

then what? Depending on how we answer question 2, the answer to 3 is either obvious or really complicated.