"ScheduleActivity" Alteration Signal Not Handled by Sequence - Incorrectly Propagates up to Invalid Ancestor

[cali-llama]

Description

This exists in Elsa 3.5.0, and was observed in 3.4, but did not exist in Elsa 3.3.

I have a workflow that uses ContinueWithIncidentStrategy. It is a Flowchart workflow where each step in the Flowchart is a Sequence of sub-step custom activities. When there is an error in one of the sub-steps and the activity faults, then we correct the underlying error and create an Alteration comprised of a ScheduleActivity to reschedule the faulted activity. This seems to work fine. We can then dispatch the workflow and all is well UNTIL we complete the Sequence for the current step of the Flowchart. At this point, we receive the error...

System.Exception: Activity CreateAndAssignTask1 is not reachable from the flowchart graph. 
    Unable to schedule it's outbound activities.    
    at Elsa.Workflows.Activities.Flowchart.Activities.Flowchart.ScheduleOutboundActivitiesAsync(FlowGraph flowGraph, FlowScope flowScope, ActivityExecutionContext flowchartContext, IActivity activity, Outcomes outcomes, Boolean completedActivityExecutedByBackwardConnection)
    at Elsa.Workflows.Activities.Flowchart.Activities.Flowchart.OnChildCompletedAsync(ActivityCompletedContext context)   
    at Elsa.Workflows.Behaviors.ScheduledChildCallbackBehavior.OnActivityCompletedAsync(ActivityCompleted signal, SignalContext context)    
    at Elsa.Workflows.Behavior.Elsa.Workflows.ISignalHandler.ReceiveSignalAsync(Object signal, SignalContext context)    
    at Elsa.Workflows.Activity.Elsa.Workflows.ISignalHandler.ReceiveSignalAsync(Object signal, SignalContext context)    
    at Elsa.Extensions.ActivityExecutionContextExtensions.SendSignalAsync(ActivityExecutionContext context, Object signal)    
    at Elsa.Workflows.ActivityExecutionContext.CompleteActivityAsync(Object result)    
    at Services.Workflow.CmxActivity.CreateAndAssignTask.ResumeBookmarkAsync(ActivityExecutionContext context) in /src/Services.Workflow/CmxActivity/CreateAndAssignTask.cs:line 175    
    at Elsa.Workflows.Middleware.Activities.DefaultActivityInvokerMiddleware.ExecuteActivityAsync(ActivityExecutionContext context)    
    at Elsa.Workflows.Runtime.Middleware.Activities.BackgroundActivityInvokerMiddleware.ExecuteActivityAsync(ActivityExecutionContext context)    
    at Elsa.Workflows.Middleware.Activities.DefaultActivityInvokerMiddleware.InvokeAsync(ActivityExecutionContext context)    
    at Elsa.Workflows.Middleware.Activities.NotificationPublishingMiddleware.InvokeAsync(ActivityExecutionContext context)    
    at Elsa.WorkflowContexts.Middleware.WorkflowContextActivityExecutionMiddleware.InvokeAsync(ActivityExecutionContext context)    
    at Elsa.Workflows.Middleware.Activities.ExecutionLogMiddleware.InvokeAsync(ActivityExecutionContext context)    
    at Elsa.Workflows.Middleware.Activities.ExceptionHandlingMiddleware.InvokeAsync(ActivityExecutionContext context)

The reason for the error arises from Flowchart.ScheduleOutboundActivitiesAsync() method, because it fails the flowGraph.IsDanglingActivity() check. That's actually expected, because what I can see happening is that the ScheduleChildActivity signal is not being handled by the parent activity, the Sequence in this case, and is instead propagating up to the Flowchart grandparent. This creates an invalid "callback" in the workflow instance state that wrecks the workflow.

You can see an example of what I mean in the workflow instance's state under "completionCallbacks"...

"completionCallbacks": {
        "$id": "8",
        "$values": [
            // ... (other callbacks excluded
            // following is the correct callback where "ownerInstanceId" is the parent Sequence activity
            {
                "$id": "11",
                "ownerInstanceId": "4541b2a5eaccb03e",
                "childNodeId": "Workflow1:Flowchart_base:Sequence_setup-1726354098792:CreateAndAssignTask1",
                "methodName": "OnChildCompleted"
            },
            // this is an invalid callback as this "ownerInstanceId" is the Flowchart grandparent activity
            {
                "$id": "12",
                "ownerInstanceId": "37352e92eea115c2",
                "childNodeId": "Workflow1:Flowchart_base:Sequence_setup-1726354098792:CreateAndAssignTask1",
                "methodName": "OnChildCompletedAsync"
            }
        ]

Note: By "wrecks", I mean that in addition to error, the workflow continues, but does not correctly stop on the next bookmark that it should (in the next Flowchart step's Sequence's first sub-step, namely, CreateAndAssignTask2).

Steps to Reproduce

Including the workflow json at ElsaFlowchartOfSequencesWorkflow.json

The scenario that I am describing has been reproduced using the activity node: Workflow1:Flowchart_base:Sequence_setup-1726354098792:CreateAndAssignTask1 where the parent is the Sequence and the grandparent is the Flowchart.

Also including the "WorkflowInstances.Data" column (state) at the point after the alteration has been run and the workflow has consumed the ScheduleChildActivity signal, but before the CreateAndAssignTask1 activity is completed (it sets a bookmark awaiting a user event).
ElsaFlowchartOfSequencesWorkflowInstance_PostAlteration_State.json

Finally, you can also see the WorkflowExecutionLogRecords here in csv format...
ElsaFlowchartOfSequencesWorkflowLogRecords.csv
You can see how after CreateAndAssignTask1 is "Completed", it gets called again and the message is the error noted above.

1. Detailed Steps:

• Execute the workflow, but trigger an exception halfway through CreateAndAssignTask1 (I'm using a manufactured exception inside the activity).
• Workflow will fault on this error and suspend itself.
• Fix the error (in my case removing the manual exception throw)
• Run the alteration that schedules the activity CreateAndAssignTask1 with the specific ActivityInstanceId.
• Alteration runs successfully, so Dispatch the workflow.
• Workflow runs CreateAndAssignTask1 until it hits the bookmark, which is expected.
• Now resume from the bookmark, which completes CreateAndAssignTask1.
• At this point we see that Sequence_setup-1726354098792 completes successfully, as expected.
• Next we see that the invalid callback is triggered inside of Flowchart.cs, which is the OnChildCompletedAsync() method. This fails the dangling activity check in ScheduleOutboundActivitiesAsync().

2. Code Snippets:
   I would think that we could solve this by simple adding a new handler in Sequence.cs like so...

public Sequence([CallerFilePath] string? source = null, [CallerLineNumber] int? line = null) : base(source, line)
{
          OnSignalReceived<BreakSignal>(OnBreakSignalReceived);
+        OnSignalReceived<ScheduleChildActivity>(OnScheduleChildActivity);
}

and then something like this for the handler implementation...

private async ValueTask OnScheduleChildActivity(ScheduleChildActivity signal, SignalContext context)
{
       var sequenceContext = context.ReceiverActivityExecutionContext;
       var childActivity = signal.Activity;

       // If the activity is not a child of this sequence, let the signal bubble up.
       if (!Activities.Contains(childActivity))
           return;

       // Prevent the signal from bubbling up.
       context.StopPropagation();

       // Schedule the activity.
       await sequenceContext.ScheduleActivityAsync(childActivity, new ScheduleWorkOptions
       {
           CompletionCallback = OnChildCompleted,
           ExistingActivityExecutionContext = signal.ActivityExecutionContext
       });
}

4. Reproduction Rate: Every time!

Expected Behavior

The workflow does not trigger CreateAndAssignTask1 after it is completed, and the workflow resumes normally as it does when no error is encountered and thus no alteration is required.

Environment

I'm running .NET9 via Mono on a MacOS Apple M2 Max ARM chip

Related Issues

I have a feeling that this issue may also be the cause of #6887

[cali-llama]

By the way, if you can think of any workaround in the meantime, that would be exceptionally helpful...what I've considered so far is to create another custom alteration to remove that invalid callback from the workflow instance state, after the ScheduleActivity alteration is complete and prior to dispatching the workflow...though not sure if that will work.

[cali-llama]

@sfmskywalker, I have confirmed that my workaround is not viable due to the WorkflowExecutionContext.CompletionCallbacks property being a ReadonlyCollection<T> with no setter. I had created a custom IAlteration to remove the invalid callback, but it does not work at runtime due to this protection.

Any suggestions would be very welcome on how to work around this issue.