Add client retry docs

Author: mostafa

.vale/Vocab/accept.txt

@@ -72,6 +72,8 @@ receiveDeadline
 receiveTimeout
 sendDeadline
 dialTimeout
+backoffMultiplier
+disableBackoffCaps
 API
 DBA
 DBRE

using-gatewayd/clients.md

@@ -40,3 +40,53 @@ You have the option to set deadlines on send and receive calls to the database s
 ## Receive timeout
 
 Since setting receive deadline kills the connection, the `receiveTimeout` property is introduced to stop the receive function from blocking the connection and waiting forever. The current value is zero, which means that it behaves like before, but one can set it to a duration string value.
+
+## Dial timeout
+
+The dial timeout is the amount of time the client should wait before giving up on dialing the database. The default value is `60s`. Setting it to `0s` disables the dial timeout.
+
+## Retries and backoff
+
+The first attempt to connect to the database server is made when the client is created, which happens instantly when GatewayD starts. However, if the first attempt fails, the client will retry the connection. The default number of retries is `3`, which means that the client will retry the connection three times before giving up. Setting it to `0` disables the retry mechanism. The client can also backoff before retrying the connection. The backoff duration is set to `1s` by default and `0s` disables the backoff mechanism. The backoff multiplier is set to `2` by default and `0` disables the backoff. The backoff multiplier is applied to the backoff duration. The backoff duration is capped at `60s` by default and the max retry is capped at `10` by default. Setting `disableBackoffCaps` to `true` disables the backoff and retry caps.
+
+{: .note }
+> The first attempt to connect to the database server is counted as a retry, hence the three retries are actually four attempts (one instant attempt and three retry attempts), and the backoff duration is applied to the second attempt and so on.
+
+The backoff duration is calculated by multiplying the backoff duration by the backoff multiplier raised to the power of the number of retries. For example, if the backoff duration is 1 second and the backoff multiplier is 2, the backoff duration will be 1 second, 2 seconds, 4 seconds, 8 seconds, etc. The backoff duration is capped at 1 minute and the backoff multiplier is capped at 10, so the backoff duration will be 1 minute after 6 retries. The backoff multiplier is capped at 10 to prevent the backoff duration from growing too quickly, unless the backoff caps are disabled. The following is the formula for calculating the backoff duration:
+
+```text
+backoff duration = backoff * (backoff multiplier ^ current retry number)
+```
+
+Considering the default values, the backoff duration will be calculated as follows:
+
+```text
+1 * 2 ^ 1 = 2 seconds
+1 * 2 ^ 2 = 4 seconds
+1 * 2 ^ 3 = 8 seconds
+```
+
+If the retries are set to `11`, the backoff duration from the fourth attempt will be calculated as follows, which is capped at 1 minute and 10 retries are made:
+
+```text
+1 * 2 ^ 4 = 16 seconds
+1 * 2 ^ 5 = 32 seconds
+1 * 2 ^ 6 = 1 minute
+1 * 2 ^ 7 = 1 minute (capped)
+1 * 2 ^ 8 = 1 minute (capped)
+1 * 2 ^ 9 = 1 minute (capped)
+1 * 2 ^ 10 = 1 minute (capped)
+```
+
+And if the backoff caps are disabled, the uncapped backoff duration for the fourth attempt will be calculated as follows:
+
+```text
+1 * 2 ^ 4 = 16 seconds
+1 * 2 ^ 5 = 32 seconds
+1 * 2 ^ 6 = 1 minute 4 seconds (uncapped)
+1 * 2 ^ 7 = 2 minutes 8 seconds (uncapped)
+1 * 2 ^ 8 = 4 minutes 16 seconds (uncapped)
+1 * 2 ^ 9 = 8 minutes 32 seconds (uncapped)
+1 * 2 ^ 10 = 17 minutes 4 seconds (uncapped)
+1 * 2 ^ 11 = 34 minutes 8 seconds (uncapped)
+```

using-gatewayd/configuration.md

@@ -83,6 +83,11 @@ clients:
     receiveTimeout: 0s # duration, 0ms/0s means no timeout
     sendDeadline: 0s # duration, 0ms/0s means no deadline
     dialTimeout: 60s # duration, 0ms/0s means no timeout
+    # Retry configuration
+    retries: 3 # 0 means no retry
+    backoff: 1s # duration
+    backoffMultiplier: 2 # 0 means no backoff
+    disableBackoffCaps: false
 
 pools:
   default:

using-gatewayd/global-configuration/clients.md

@@ -25,6 +25,10 @@ GatewayD supports multiple client configurations. Each client in each configurat
 | receiveTimeout     | duration (string) | 0s             | Valid duration strings | The amount of time the client should wait before giving up on call to receive. `0s` disables receive timeout.         |
 | sendDeadline       | duration (string) | 0s             | Valid duration strings | The amount of time the client should wait before giving up on call to send. `0s` disables send deadline.              |
 | dialTimeout        | duration (string) | 60s            | Valid duration strings | The amount of time the client should wait before giving up on dialing the database. `0s` disables dial timeout.       |
+| retries            | number            | 3              | Positive integers      | The amount of times to retry a failed connection. `0` means no retry.                                                 |
+| backoff            | duration (string) | 1s             | Valid duration strings | The amount of time to wait before retrying a failed connection. `0s` means no backoff.                                |
+| backoffMultiplier  | number            | 2              | Positive integers      | The multiplier to apply to the backoff duration. `0` means no backoff.                                                |
+| disableBackoffCaps | boolean           | False          | True, False            | Whether to disable the backoff caps for backoff multiplier and backoff duration.                                      |
 
 ```yaml
 clients:
@@ -38,4 +42,9 @@ clients:
     receiveTimeout: 0s # duration, 0ms/0s means no timeout
     sendDeadline: 0s # duration, 0ms/0s means no deadline
     dialTimeout: 60s # duration, 0ms/0s means no timeout
+    # Retry configuration
+    retries: 3 # 0 means no retry
+    backoff: 1s # duration
+    backoffMultiplier: 2 # 0 means no backoff
+    disableBackoffCaps: false
 ```