Workflow stuck at Pending state when unhandled exception happen on FlowChartSerialization

[regan32]

Description

When error occured at serialization process(I Think there is a same behaviour for other unhandled exceptions). For my case it is OutOfMemoryException. Workflow stucks at pending state.
OutOfMemoryException stacktrace:
at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags) at System.Text.Json.Utf8JsonWriter.WriteStringEscapeValue(ReadOnlySpan1 value, Int32 firstEscapeIndexVal) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at Elsa.Workflows.Serialization.Converters.PolymorphicObjectConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) at Elsa.Workflows.Serialization.Converters.PolymorphicDictionaryConverter.Write(Utf8JsonWriter writer, IDictionary2 value, JsonSerializerOptions options) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at System.Text.Json.JsonSerializer.WriteStringAsObject(Object value, JsonTypeInfo jsonTypeInfo) at Elsa.Workflows.Serialization.Converters.PolymorphicObjectConverter.Write(Utf8JsonWriter writer, Object value, JsonSerializerOptions options) at Elsa.Workflows.Serialization.Converters.PolymorphicDictionaryConverter.Write(Utf8JsonWriter writer, IDictionary2 value, JsonSerializerOptions options) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonPropertyInfo1.GetMemberAndWriteJson(Object obj, WriteStack& state, Utf8JsonWriter writer) at System.Text.Json.Serialization.Converters.ObjectDefaultConverter1.OnTryWrite(Utf8JsonWriter writer, T value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.TryWrite(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.JsonConverter1.WriteCore(Utf8JsonWriter writer, T& value, JsonSerializerOptions options, WriteStack& state) at System.Text.Json.Serialization.Metadata.JsonTypeInfo1.Serialize(Utf8JsonWriter writer, T& rootValue, Object rootValueBoxed) at System.Text.Json.JsonSerializer.WriteString[TValue](TValue& value, JsonTypeInfo1 jsonTypeInfo) at Elsa.EntityFrameworkCore.Modules.Management.EFCoreWorkflowInstanceStore.OnSaveAsync(ManagementElsaDbContext managementElsaDbContext, WorkflowInstance entity, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Store2.SaveAsync(TEntity entity, Expression1 keySelector, Func4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Store2.SaveAsync(TEntity entity, Expression1 keySelector, Func4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Store2.SaveAsync(TEntity entity, Expression1 keySelector, Func4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.EntityStore2.SaveAsync(TEntity entity, Func4 onSaving, CancellationToken cancellationToken) at Elsa.EntityFrameworkCore.Modules.Management.EFCoreWorkflowInstanceStore.SaveAsync(WorkflowInstance instance, CancellationToken cancellationToken) at Elsa.Workflows.Management.Services.WorkflowInstanceManager.SaveAsync(WorkflowInstance workflowInstance, CancellationToken cancellationToken) at Elsa.Workflows.Management.Services.WorkflowInstanceManager.SaveAsync(WorkflowState workflowState, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.DefaultCommitStateHandler.CommitAsync(WorkflowExecutionContext workflowExecutionContext, WorkflowState workflowState, CancellationToken cancellationToken) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowExecutionContext workflowExecutionContext) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowGraph workflowGraph, WorkflowState workflowState, RunWorkflowOptions options, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(WorkflowInstance workflowInstance, RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.<>c__DisplayClass8_0.<<RunInstanceAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.BookmarkResumer.ResumeAsync(BookmarkFilter filter, ResumeBookmarkOptions options, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.StoreBookmarkQueue.EnqueueAsync(NewBookmarkQueueItem item, CancellationToken cancellationToken) at Paragon.LowCode.Workflows.Elsa3.ElsaActivities.Handlers.ResumeDispatchWorkflowActivity.HandleAsync(WorkflowExecuted notification, CancellationToken cancellationToken) in /_/src/Paragon.LowCode.Workflows.Elsa3.ElsaActivities/Handlers/ResumeDispatchWorkflowActivity.cs:line 74 at Elsa.Mediator.PublishingStrategies.SequentialProcessingStrategy.PublishAsync(NotificationStrategyContext context) at Elsa.Mediator.Middleware.Notification.Components.NotificationHandlerInvokerMiddleware.InvokeAsync(NotificationContext context) at Elsa.Mediator.Middleware.Notification.NotificationPipeline.ExecuteAsync(NotificationContext context) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, IEventPublishingStrategy strategy, CancellationToken cancellationToken) at Elsa.Mediator.Services.DefaultMediator.SendAsync(INotification notification, CancellationToken cancellationToken) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowExecutionContext workflowExecutionContext) at Elsa.Workflows.WorkflowRunner.RunAsync(WorkflowGraph workflowGraph, WorkflowState workflowState, RunWorkflowOptions options, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(WorkflowInstance workflowInstance, RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.LocalWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.<>c__DisplayClass8_0.<<RunInstanceAsync>b__0>d.MoveNext() --- End of stack trace from previous location --- at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.WithLockAsync[R](Func1 func) at Elsa.Workflows.Runtime.Distributed.DistributedWorkflowClient.RunInstanceAsync(RunWorkflowInstanceRequest request, CancellationToken cancellationToken) at Elsa.MassTransit.Consumers.DispatchWorkflowRequestConsumer.DispatchExistingWorkflowInstanceAsync(DispatchWorkflowDefinition message, CancellationToken cancellationToken) at Elsa.MassTransit.Consumers.DispatchWorkflowRequestConsumer.Consume(ConsumeContext1 context) at MassTransit.DependencyInjection.ScopeConsumerFactory1.Send[TMessage](ConsumeContext1 context, IPipe1 next) in /_/src/MassTransit/DependencyInjection/DependencyInjection/ScopeConsumerFactory.cs:line 22 at MassTransit.DependencyInjection.ScopeConsumerFactory1.Send[TMessage](ConsumeContext1 context, IPipe1 next) in //src/MassTransit/DependencyInjection/DependencyInjection/ScopeConsumerFactory.cs:line 22 at MassTransit.Middleware.ConsumerMessageFilter2.MassTransit.IFilter<MassTransit.ConsumeContext<TMessage>>.Send(ConsumeContext1 context, IPipe1 next) in /_/src/MassTransit/Middleware/ConsumerMessageFilter.cs:line 48 at MassTransit.Middleware.ConsumerMessageFilter2.MassTransit.IFilter<MassTransit.ConsumeContext>.Send(ConsumeContext1 context, IPipe1 next) in //src/MassTransit/Middleware/ConsumerMessageFilter.cs:line 73 at Elsa.MassTransit.Middleware.TenantConsumeMiddleware1.Send(ConsumeContext1 context, IPipe1 next) at Elsa.MassTransit.Middleware.TenantConsumeMiddleware1.Send(ConsumeContext1 context, IPipe1 next) at MassTransit.Middleware.ScopedConsumeFilter2.Send(ConsumeContext1 context, IPipe1 next) in /_/src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.ScopedConsumeFilter2.Send(ConsumeContext1 context, IPipe1 next) in //src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.ScopedConsumeFilter2.Send(ConsumeContext1 context, IPipe1 next) in /_/src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.ScopedConsumeFilter2.Send(ConsumeContext1 context, IPipe1 next) in //src/MassTransit/Middleware/ScopedConsumeFilter.cs:line 26 at MassTransit.Middleware.InMemoryOutboxFilter2.Send(TContext context, IPipe1 next) in //src/MassTransit/Middleware/InMemoryOutboxFilter.cs:line 35 at MassTransit.Middleware.InMemoryOutboxFilter2.Send(TContext context, IPipe1 next) in //src/MassTransit/Middleware/InMemoryOutboxFilter.cs:line 45 at MassTransit.Middleware.TeeFilter1.<>c__DisplayClass5_0.<<Send>g__SendAsync|1>d.MoveNext() in /_/src/MassTransit/Middleware/TeeFilter.cs:line 40 --- End of stack trace from previous location --- at MassTransit.Middleware.ConsumeContextOutputMessageTypeFilter1.SendToOutput(IPipe1 next, ConsumeContext1 pipeContext) in //src/MassTransit/Middleware/ConsumeContextOutputMessageTypeFilter.cs:line 76 at MassTransit.Middleware.ConsumeContextOutputMessageTypeFilter1.SendToOutput(IPipe1 next, ConsumeContext1 pipeContext) in /_/src/MassTransit/Middleware/ConsumeContextOutputMessageTypeFilter.cs:line 108 at MassTransit.Middleware.ConsumeContextMessageTypeFilter.<>c__DisplayClass8_0.<<Send>g__SendAsync|0>d.MoveNext() in /_/src/MassTransit/Middleware/MessageTypeFilter.cs:line 76 --- End of stack trace from previous location --- at MassTransit.Middleware.DeserializeFilter.Send(ReceiveContext context, IPipe1 next) in //src/MassTransit/Middleware/DeserializeFilter.cs:line 40 at MassTransit.Middleware.RescueFilter2.MassTransit.IFilter<TContext>.Send(TContext context, IPipe1 next) in /_/src/MassTransit/Middleware/RescueFilter.cs:line 43`

Steps to Reproduce

I can't post my workflow here, because it's realy huge with customized activities.
Best way to repoduce issue is simulate unhandled exception at FlowchartJsonConverter while saving FlowChart at WorkflowInstance startup

6. Additional Configuration:
   • MassTransit is used for jobs queue

Expected Behavior

Workflow goes to Failed state

Actual Behavior

Workflow stuck at Pending state

Screenshots

If possible, add screenshots or screen recordings to help explain the problem.

Environment

• Elsa 3.4
• Ubuntu 22.04.

Related Issues

Link to any related issues here.

[sfmskywalker]

Does this happen also with Elsa 3.5.0?

[regan32]

Right now i can't check it, some of my SetVariable activities doesn't work at 3.5.0. Seems like something changed at JS syntax