Merge commit '4b172b60ee5ac3667affca647bd13a0cba71ccc2' into named-side-effects

Author: slinkydeveloper

sdk-core/src/main/service-protocol/dev/restate/service/protocol.proto

@@ -75,6 +75,14 @@ message ErrorMessage {
   string message = 2;
   // Contains a verbose error description, e.g. the exception stacktrace.
   string description = 3;
+
+  // Entry that caused the failure. This may be outside the current stored journal size.
+  // If no specific entry caused the failure, the current replayed/processed entry can be used.
+  uint32 related_entry_index = 4;
+  // Name of the entry that caused the failure. Empty if no name was set.
+  string related_entry_name = 5;
+  // Entry type. 0 if unknown.
+  uint32 related_entry_type = 6;
 }
 
 // Type: 0x0000 + 4
@@ -96,7 +104,7 @@ message EndMessage {
 //   * bytes value = 14 for carrying the result value
 //   * Failure failure = 15 for carrying a failure
 //
-// The tag numbers 13, 14 and 15 are reserved and shouldn't be used for other fields.
+// The tag numbers 12, 13, 14 and 15 are reserved and shouldn't be used for other fields.
 
 // ------ Input and output ------
 
@@ -107,6 +115,9 @@ message InputEntryMessage {
   repeated Header headers = 1;
 
   bytes value = 14;
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: No
@@ -117,6 +128,9 @@ message OutputEntryMessage {
     bytes value = 14;
     Failure failure = 15;
   };
+
+  // Entry name
+  string name = 12;
 }
 
 // ------ State access ------
@@ -132,6 +146,9 @@ message GetStateEntryMessage {
     bytes value = 14;
     Failure failure = 15;
   };
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: No
@@ -140,19 +157,27 @@ message GetStateEntryMessage {
 message SetStateEntryMessage {
   bytes key = 1;
   bytes value = 3;
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: No
 // Fallible: No
 // Type: 0x0800 + 2
 message ClearStateEntryMessage {
   bytes key = 1;
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: No
 // Fallible: No
 // Type: 0x0800 + 3
 message ClearAllStateEntryMessage {
+  // Entry name
+  string name = 12;
 }
 
 // Completable: Yes
@@ -167,6 +192,9 @@ message GetStateKeysEntryMessage {
     StateKeys value = 14;
     Failure failure = 15;
   };
+
+  // Entry name
+  string name = 12;
 }
 
 // ------ Syscalls ------
@@ -183,6 +211,9 @@ message SleepEntryMessage {
     Empty empty = 13;
     Failure failure = 15;
   }
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: Yes
@@ -203,6 +234,9 @@ message InvokeEntryMessage {
     bytes value = 14;
     Failure failure = 15;
   };
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: No
@@ -224,6 +258,9 @@ message BackgroundInvokeEntryMessage {
 
   // If this invocation has a key associated (e.g. for objects and workflows), then this key is filled in. Empty otherwise.
   string key = 6;
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: Yes
@@ -235,6 +272,9 @@ message AwakeableEntryMessage {
     bytes value = 14;
     Failure failure = 15;
   };
+
+  // Entry name
+  string name = 12;
 }
 
 // Completable: No
@@ -248,6 +288,9 @@ message CompleteAwakeableEntryMessage {
     bytes value = 5;
     Failure failure = 6;
   };
+
+  // Entry name
+  string name = 12;
 }
 
 // --- Nested messages

sdk-core/src/main/service-protocol/service-invocation-protocol.md

@@ -11,7 +11,7 @@ The system is composed of two actors:
   - SDK, which contains the implementation of the Restate Protocol
   - User business logic, which interacts with the SDK to access Restate system calls (or syscalls)
 
-Each service method invocation is modeled by the protocol as a state machine, where state transitions can be caused
+Each invocation is modeled by the protocol as a state machine, where state transitions can be caused
 either by user code or by _Runtime events_.
 
 Every state transition is logged in the _Invocation journal_, used to implement Restate's durable execution model. The
@@ -32,11 +32,12 @@ The state machine is summarized in the following diagram:
 ```mermaid
 sequenceDiagram
     Note over Runtime,SDK: Start
-    Runtime->>SDK: HTTP Request to /invoke/{service}/{method}
+    Runtime->>SDK: HTTP Request to /invoke/{service}/{handler}
     Runtime->>SDK: StartMessage
     Note over Runtime,SDK: Replaying
     Runtime->>SDK: [...]EntryMessage(s)
     Note over Runtime,SDK: Processing
+    SDK->>Runtime: HTTP Response headers
     loop
         SDK->>Runtime: [...]EntryMessage
         Runtime->>SDK: CompletionMessage and/or EntryAckMessage
@@ -102,9 +103,7 @@ protocol mandates the following messages:
 
 ### Message stream
 
-In order to execute a service method invocation, service deployment and restate Runtime open a single stream between the
-runtime and the service deployment. Given 10 concurrent service method invocations to a service deployment, there are 10
-concurrent streams, each of them mapping to a specific invocation.
+In order to execute an invocation, service deployment and restate Runtime open a single stream between the runtime and the service deployment. Given 10 concurrent invocations to a service deployment, there are 10 concurrent streams, each of them mapping to a specific invocation.
 
 Every unit of the stream contains a Message serialized using the
 [Protobuf encoding](https://protobuf.dev/programming-guides/encoding/), using the definitions in
@@ -119,10 +118,23 @@ in two modes:
   runtime. Once the service deployment starts sending messages to the runtime, the runtime cannot send messages anymore
   back to the service deployment.
 
-When opening the stream, the request method MUST be `POST` and the request path MUST have the following format:
+A message stream MUST start with `StartMessage` and MUST end with either:
+
+- One [`SuspensionMessage`](#suspension)
+- One [`ErrorMessage`](#failures)
+- One `EndMessage`
+
+If the message stream does not end with any of these two messages, it will be considered equivalent to sending an
+`ErrorMessage` with an [unknown failure](#failures).
+
+The `EndMessage` marks the end of the invocation lifecycle, that is the end of the journal.
+
+### Initiating the stream
+
+When opening the stream, the HTTP request method MUST be `POST` and the request path MUST have the following format:
 
 ```
-/invoke/{fullyQualifiedServiceName}/{methodName}
+/invoke/{serviceName}/{handlerName}
 ```
 
 For example:
@@ -133,19 +145,29 @@ For example:
 
 An arbitrary path MAY prepend the aforementioned path format.
 
-In case the path format is not respected, or `fullyQualifiedServiceName` or `methodName` is unknown, the SDK MUST close
-the stream replying back with a `404` status code.
+In case the path format is not respected, or `serviceName` or `handlerName` is unknown, the SDK MUST close the stream replying back with a `404` status code.
 
-A message stream MUST start with `StartMessage` and MUST end with either:
+In case the invocation is accepted, `200` status code MUST be returned.
 
-- One [`SuspensionMessage`](#suspension)
-- One [`ErrorMessage`](#failures)
-- One `EndMessage`
+Additionally, the header `x-restate-user-agent` MAY be sent back, with the following format:
 
-If the message stream does not end with any of these two messages, it will be considered equivalent to sending an
-`ErrorMessage` with an [unknown failure](#failures).
+```http request
+x-restate-user-agent: <sdk-name> / <sdk-version>; <additional-metadata?>
+```
 
-The `EndMessage` marks the end of the invocation lifecycle, that is the end of the journal.
+E.g.:
+
+```http request
+x-restate-user-agent: restate-sdk-java/0.8.0
+```
+
+Or:
+
+```http request
+x-restate-user-agent: restate-sdk-java/0.8.0; gitHash=0c5917b
+```
+
+This header is used for observability purposes by the Restate observability tools.
 
 ### Message header
 
@@ -275,6 +297,11 @@ index of the corresponding entry.
     |                             Length                            |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 
+#### Entry names
+
+Every Journal entry has a field `string name = 12`, which can be set by the SDK when recording the entry. This field is
+used for observability purposes by Restate observability tools.
+
 ### Journal entries reference
 
 The following tables describe the currently available journal entries. For more details, check the protobuf message
@@ -365,6 +392,11 @@ additional features to the users.
 The protocol allows the SDK to register an arbitrary entry type within the journal. The type MUST be `>= 0xFC00`. The
 runtime will treat this entry as any other entry, persisting it and sending it during replay in the correct order.
 
+Custom entries MAY have the entry name field `12`, as described in [entry names](#entry-names).
+
+The field numbers 13, 14 and 15 MUST not be used, as they're reserved for completable journal entries, as described in
+[completable journal entries](#completable-journal-entries-and-completionmessage).
+
 **Header**
 
     0                   1                   2                   3
@@ -405,3 +437,15 @@ A possible implementation could be the following. Given a user requests a state
 
 In order for the aforementioned algorithm to work, set, clear and clear all state operations must be reflected on the
 local `state_map` as well.
+
+### Side effect/Run
+
+The side effect/run feature allows users to execute arbitrary non-deterministic code within their service and record the
+result, such that on re-executions the stored value will be used instead of replaying the given code.
+
+SDKs MAY implement the side effect feature by storing the result using a custom entry message, as described in
+[Custom entry](#custom-entry-messages). By convention, the SDKs SHOULD use the entry type `FC01` for side effects.
+
+When storing side effects, SDKs MAY need to wait for the
+[acknowledgment of the stored entry](#acknowledgment-of-stored-entries) before continuing the execution of the
+invocation.