WG - Unified Saga implementation

[xstefank]

Objective:

The main objective of this working group is to unify Quarkus's story about Saga-based transactions.

The Problem

The primary saga solution for distributed transactions is narayana-lra extension. However, it wasn't maintained and marketed as the number one solution for distributed sagas. So we either need to update LRA to make a unified solution for Quarkus or pick a different solution that was introduced in the meantime. This WG will unify Quarkus's story in a distributed transaction solution.

The proposed Solution

We will investigate and compare available solutions for the saga-based implementation of the distributed tx processing. This will be documented, and the choice should be proven to be able to drive development and innovation in this area. We will ensure the solution meets the Quarkus standards (build/runtime split, dev service, dev ui).

Definition of Done

• A new guide for the distributed tx (or the chosen solution if there is only one)
• Blog post comparing available solutions or just marketing the chosen solution
• Quarkus insights about this topic

Scope of Work

The scope of this WG is to pick a chosen saga-based implementation and make it Quarkus native (dev mode, any possible enhancements). We will choose implementation with an already available Quarkus integration. Research of new saga-based alternatives is out of scope.

Organizing the Work

Communication and Transparency:

Coordination will be done in this GitHub discussion and the individual GitHub issues. GitHub issues can be closed if the saga-solution is not to be updated anymore. The decision of the chosen solution will be stated also as the resolution of #34181.

This work can take place in either Quarkus core (narayana-lra) or some new extension that will be provided in Quarkiverse.

Point of Contact:

Martin Stefanko

Expected Timeline:

Milestone 1 - Document in this discussion saga-based implementations alternatives
Milestone 2 - Pick one solution that will be Quarkus Saga solution
Milestone 3 - Update the chosen alternative for Quarkus standards (simplification of use, dev * features)
Milestone 4 - Create a guide to document it
Milestone 5 - Blog + Quarkus insights

Potential Outcomes & Deliverables

• Guide for distributed tx (including non-saga based solutions if there are some)
• Blog
• Quarkus insights

Existing issues

The following issues should be resolved/tackled in the scope of this working group:

• Improve / Define distributed transaction support #34181
• [Extension Proposal] Auto-compensating saga extension #40588
• Provide a way to specify default extension dependency in case the user won't provide any of required options #28809
• Default Quarkus application generated by maven plugin with narayana-lra extension doesn't compile #28744
• DevService for LRA #23558
• Implement Dev Services for the LRA coordinator #18425

[marcosgopen]

I am joining the working group +1

[cescoffier]

Awesome! Are you ready to start the WG? If so, I will bootstrap the setup.

My main question is: Should it be in Quarkus' main org or Quarkiverse? I can see a few action items that would deserve new Quarkiverse extensions.

[xstefank]

Yes, we are ready. And it depends, Narayana LRA is in core repo, everything else in Quarkiverse. From the preliminary investigation it seems that LRA might still be the way forward and most of the issues are for LRA. So, I would scope it under the main repo.

[cescoffier]

Board created: WG - Unified Saga implementation

[pcheucle]

I am joining this working group 😃

[xstefank]

Thanks @marcosgopen and @pcheucle. If any of you would have time for "Milestone 1 - Document in this discussion saga-based implementations alternatives" to do in this issue, please go ahead. I can't allocate enough time for this right now.

[xstefank]

Milestone 1 - Document in this discussion saga-based implementations alternatives

The purpose of this comment is to provide an overview of the current available solutions. If there are more, please feel free to ping me here or elsewhere, and I will add them here.

Option 1: Narayana Long Running Actions (LRA)

Narayana LRA is the only core extension that has been merged into the Quarkus repository. It is the main implementation of the MicroProfile LRA specification (GitHub).

It supports the orchestration-based saga pattern execution through the provided LRA coordinator, which is provided as a Quarkus REST application at https://quay.io/repository/jbosstm/lra-coordinator.

All communication, both with the coordinator and individual LRA participants, must be executed through REST/HTTP. This is a limitation of the specification, but there is no technical reason why Narayana LRA couldn't support different protocols.

Users annotate their REST resources with annotations provided by the MP LRA specification. When the REST method annotated with @LRA is called, the extension automatically registers this REST resource as an LRA Participant with a REST call to the LRA coordinator. To propagate the LRA/saga to different services, the caller must specify an HTTP header that provides the LRA ID (saga context), which is then used in the same manner in the called service. The extension/library can automatically add this context header, so users don't have to specify it manually.

The extension code is at https://github.com/quarkusio/quarkus/tree/main/extensions/narayana-lra, and the library code is available at https://github.com/jbosstm/lra. The LRA coordinator project code is at https://github.com/jbosstm/lra-coordinator-quarkus.

The API is annotation-driven.

Advantages

• Official MicroProfile specification
• Implementation support from the Narayana team in Red Hat/IBM
• No vendor lock-in/ability to change the implementation (if it is specification compatible)

Disadvantages

• No active development of new features, only maintained as of now
• Missing features (e.g., Dev services)
• Limited to the REST/HTTP
• LRA coordinator is a SPoF

Summary

Narayana LRA	-
Extension	https://github.com/quarkusio/quarkus/tree/main/extensions/narayana-lra
Library	https://github.com/jbosstm/lra
Specification	https://github.com/microprofile/microprofile-lra

Option 2: Temporal

Temporal is a commercial, complete end-to-end solution - https://temporal.io/. It provides a single package for enterprise functionality bundles on top of the saga pattern, including monitoring, retries, and timers. It is open-source and available as a Docker image for self-hosted deployment. It also offers a paid SaaS solution called Temporal Cloud, which provides additional features such as high availability.

It provides a custom SDK available in multiple languages, including Java. The Quarkiverse extension provides the Quarkus integration (CDI integration, native) on top of the Java SDK that is based mainly on static methods.

The architecture consists of a workflow that defines the saga, activities that correspond to individual actions within the Workflow, and Workers that are the activity executors. The Temporal SDK communicates with the Temporal Service (either self-hosted or cloud-based) to manage and execute the Workflow defined in the Java method. Different workers can perform various activities. Each worker polls the Temporal Service to check if tasks have been assigned to them.

Temporal provides many enterprise features that ease the development of sagas. To name a few, Web UI, CLI, workflow replays, and durable logs. The Workflow method executes in steps by individual activities. It is executed continuously throughout the entire workflow execution, which can take even days.

The API is based on the implementation of the Workflow interface method that defines individual steps.

Advantages

• Built-in enterprise features (monitoring, retries, delays)
• Web UI / graphical view of the saga execution
• Easy switch between self-hosted and cloud versions

Disadvantages

• Full feature set requires payment (Temporal Cloud)
• Free solution needs a lot of manual adjustments (DB, monitoring)
• Proprietary SDK and vendor lock-in

Summary

Quarkus Temporal	-
Extension	https://github.com/quarkiverse/quarkus-temporal/tree/main
Library	https://github.com/temporalio/sdk-java
Site	https://temporal.io/

Option 3: Saga CQRS

Solution proposed by @xian75 that focuses on databases. Although it isn't well-documented yet, I gathered some information from the comments in #40588 and the provided code and examples.

The idea is to combine CQRS with the saga pattern to achieve data consistency. While it is an interesting idea, the saga pattern has much broader applicability; therefore, we can't consider this as a final Quarkus saga solution.

The Orchestrator executes the transaction in two phases. In the first phase, it asks the participants to mark the DB rows they want to add/change/delete by executing a local ACID transaction to change the "status" column of their tables, which requires that the participant entities contain this specific column (they have to extend the entity provided by the library). If all participants mark their changes successfully, tx moves to commit status; otherwise, it rolls back. In the second phase, if the transaction is in the commit status, the orchestrator executes a resolution loop that attempts to execute the marked operation in the participant tables with each iteration (again in a new local ACID tx), iterating until all operations are complete. If the outcome of the first phase is rollback, the markings are rolled back.

The API requires users to extend specific entities and REST clients to plug into the expected processing.

Advantages

• Focus on the data integrity

Disadvantages

• limited to the database integration
• not documented, seems that it still requires some work
• very tight coupling to the implementation and the need to adjust the user data model
• REST/HTTP restricted

Summary

Saga CQRS	-
Extension	https://github.com/xian75/sagacqrs/

[pcheucle]

@xstefank I can take Temporal, I will have some time in August.

[xstefank]

Thanks @pcheucle please adjust what I will create today. I'm reading through docs and experimenting this morning.