docs: Document HTTP transport for the SQL API + describe transports in the REST API (#9616)

Author: igorlukanin

docs/pages/product/apis-integrations/recipes/_meta.js

@@ -3,5 +3,6 @@ module.exports = {
   "cast-numerics": "Numeric values",
   "sorting": "Custom sorting",
   "pagination": "Pagination",
-  "drilldowns": "Drilldowns"
+  "drilldowns": "Drilldowns",
+  "real-time-data-fetch": "Real-time data fetch"
 }

docs/pages/product/apis-integrations/recipes/real-time-data-fetch.mdx

@@ -0,0 +1,93 @@
+# Real-Time data fetch in the REST API
+
+## Use case
+
+When building an embedded analytics application, you'd like to provide a real-time
+experience to the users. On some page, you'd like to display a chart that updates
+as soon as the data changes in the database.
+
+## Configuration
+
+When using the [REST API][ref-rest-api], you can use the [WebSocket
+transport][ref-websocket-transport] to receive real-time updates. Using this
+transport enables subscriptions to real-time updates.
+
+### Client code
+
+JavaScript example:
+
+```javascript
+import cube from "@cubejs-client/core";
+import WebSocketTransport from "@cubejs-client/ws-transport";
+
+const cubeApi = cube({
+  transport: new WebSocketTransport({
+    authorization: CUBE_TOKEN,
+    apiUrl: "ws://localhost:4000/",
+  }),
+});
+
+// Create a subscription
+const subscription = cubeApi.subscribe(
+  {
+    measures: ["logs.count"],
+    timeDimensions: [
+      {
+        dimension: "logs.time",
+        granularity: "hour",
+        dateRange: "last 1440 minutes",
+      },
+    ],
+  },
+  options,
+  (error, resultSet) => {
+    if (!error) {
+      // handle the update
+    }
+  }
+);
+
+// Later on, unsubscribe from subscription
+subscription.unsubscribe();
+```
+
+React example:
+
+```javascript
+import { useCubeQuery } from "@cubejs-client/react";
+
+const Chart = ({ query }) => {
+  const { resultSet, error, isLoading } = useCubeQuery(query, {
+    // The component will automatically unsubscribe when unmounted
+    subscribe: true,
+  });
+
+  if (isLoading) {
+    return <div>Loading...</div>;
+  }
+
+  if (error) {
+    return <pre>{error.toString()}</pre>;
+  }
+
+  if (!resultSet) {
+    return null;
+  }
+
+  return <LineChart resultSet={resultSet} />;
+};
+```
+
+### Refresh rate
+
+Real-time data fetch obeys the [`refresh_key`][ref-refresh-key].
+In order to provide a desired refresh rate, `refresh_key` should reflect the rate of
+change of the underlying data set; the querying time should also be much less than the
+desired refresh rate. Please use the [`every`][ref-every] parameter to adjust the
+refresh interval.
+
+
+[ref-rest-api]: /product/apis-integrations/rest-api
+[ref-websocket-transport]: /product/apis-integrations/rest-api#websocket-transport
+[ref-refresh-key]: /product/caching#refresh-keys
+[ref-every]: /product/data-modeling/reference/cube#refresh_key

docs/pages/product/apis-integrations/rest-api.mdx

@@ -33,7 +33,16 @@ REST API also provides endpoints for [GraphQL API][ref-graphql-api] and
 [Orchestration API][ref-orchestration-api]. However, they target specific use
 cases and are not considered part of the REST API.
 
-## Example request
+## Transport
+
+The REST API supports the following transports:
+
+| Transport | Description | When to use |
+| --- | --- | --- |
+| HTTP | HTTP-based protocol with long polling | Use by default |
+| WebSocket | [WebSocket][link-websocket]-based protocol with support for real-time updates | Use for applications that require subscriptions to real-time updates |
+
+### HTTP transport
 
 You can use the `curl` utility to execute requests to the REST API:
 
@@ -88,6 +97,36 @@ curl \
   http://localhost:4000/cubejs-api/v1/meta | jq
 ```
 
+You can also use the [JavaScript SDK][ref-javascript-sdk] to call the REST API
+from your JavaScript applications.
+
+### WebSocket transport
+
+WebSocket can be used to provide real-time experience. You can configire it via the
+`CUBEJS_WEB_SOCKETS` environment variable.
+
+Clients using the [JavaScript SDK][ref-javascript-sdk] can be switched to WebSocket
+by passing `WebSocketTransport` to the `CubeApi` constructor:
+
+```javascript
+import cube from "@cubejs-client/core"
+import WebSocketTransport from "@cubejs-client/ws-transport"
+
+const cubeApi = cube({
+  transport: new WebSocketTransport({
+    authorization: CUBE_TOKEN,
+    apiUrl: "ws://localhost:4000/"
+  })
+})
+```
+
+<ReferenceBox>
+
+See [this recipe][ref-recipe-real-time-data-fetch] for an example of real-time
+data fetch using the WebSocket transport.
+
+</ReferenceBox>
+
 ## Configuration
 
 REST API is enabled by default and secured using [API scopes][self-api-scopes]
@@ -188,30 +227,7 @@ how to generate it.
 In the development environment the token is not required for authorization, but
 you can still use it to pass a security context.
 
-### Continue wait
-
-If the request takes too long to be processed, Cube Backend responds with
-`{ "error": "Continue wait" }` and 200 status code. This is how the long polling
-mechanism in Cube is implemented. Clients should continuously retry the same
-query in a loop until they get a successful result. Subsequent calls to the Cube
-endpoints are idempotent and don't lead to scheduling new database queries if
-not required by the [`refresh_key`][ref-schema-ref-cube-refresh-key]. Also,
-receiving `Continue wait` doesn't mean the database query has been canceled, and
-it's actually still being processed by the Cube. Database queries that weren't
-started and are no longer waited by the client's long polling loop will be
-marked as orphaned and removed from the querying queue.
-
-Possible reasons of **Continue wait**:
-
-- The query requested is heavy, and it takes some time for the database to
-  process it. Clients should wait for its completion, continuously sending the
-  same REST API request. [`continueWaitTimeout`][ref-conf-queue-opts] can be
-  adjusted in order to change the time Cube waits before returning
-  `Continue wait` message.
-- There are many queries requested and Cube backend queues them to save database
-  from overloading.
-
-### Error Handling
+### Error handling
 
 Cube REST API has basic errors and HTTP Error codes for all requests.
 
@@ -222,7 +238,7 @@ Cube REST API has basic errors and HTTP Error codes for all requests.
 | 403    | Invalid token                  | The auth token provided is not valid. It may be expired or have invalid signature.                   |
 | 500    | Error message                  | Cube internal server error. Check error message for details.                                         |
 
-### Request Span Annotation
+### Request span annotation
 
 For monitoring tools such as Cube Cloud proper request span annotation should be
 provided in `x-request-id` header of a request. Each request id should consist
@@ -232,27 +248,31 @@ should be unique for each separate request. `spanId` should define user
 interaction span such us `Continue wait` retry cycle and it's value shouldn't
 change during one single interaction.
 
-### Pagination
-
-Cube supports paginated requests for the `/v1/load` endpoint by including
-[`limit` and `offset` parameters][ref-rest-query-format] in the query. For
-example, the following query will retrieve rows 101-200 from the `Orders` cube:
-
-```json
-{
-  "dimensions": ["Orders.status"],
-  "measures": ["Orders.count"],
-  "timeDimensions": [
-    {
-      "dimension": "Orders.createdAt",
-      "dateRange": "last year",
-      "granularity": "day"
-    }
-  ],
-  "limit": 100,
-  "offset": 100
-}
-```
+## Troubleshooting
+
+### `Continue wait`
+
+If the request takes too long to be processed, the REST API responds with
+`{ "error": "Continue wait" }` and the status code 200.
+
+This is part of the long polling mechanism:
+- After an API instance receives a request, it adds the query to the query queue.
+- If the query takes too long to be processed, the API instance will respond with `Continue wait`. Receiving `Continue wait` doesn't mean the database query has been canceled, and it's actually still being processed by Cube.
+- Clients who received `Continue wait` should continuously retry the same query in a loop until they get a successful result. Database queries that are no longer retried by clients will be marked as orphaned and removed from the query queue. Subsequent calls to the REST API are idempotent and don't lead to scheduling new database queries if not required by the [`refresh_key`][ref-schema-ref-cube-refresh-key].
+
+Possible reasons of `Continue wait`:
+- The query is too heavy for the upstream database, i.e., it takes too long to be processed.
+- The maximum [concurrency][ref-concurrency] is reached, i.e., there are many queries
+waiting in the query queue until the previous ones are completed.
+
+[`continueWaitTimeout`][ref-conf-queue-opts] configuration option can be adjusted
+in order to change the time Cube waits before returning `Continue wait` message.
+However, it's recommended to address the root cause of the issue instead of
+increasing the timeout:
+- Switch from a [traditional database][ref-traditional-databases] to a [data
+warehouse][ref-data-warehouses].
+- Increase the [concurrency][ref-concurrency] if your database can handle it.
+- Implement [pre-aggregations][ref-pre-aggregations] to reduce the query time by using Cube Store.
 
 [mdn-cors]: https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS
 [ref-config-js]: /product/configuration/reference/config
@@ -286,4 +306,11 @@ example, the following query will retrieve rows 101-200 from the `Orders` cube:
 [self-cors]: #configuration-cors
 [ref-ref-rest-api]: /product/apis-integrations/rest-api/reference
 [link-jq-utility]: https://jqlang.github.io/jq/
-[gh-cube-openspec]: https://github.com/cube-js/cube/blob/master/packages/cubejs-api-gateway/openspec.yml
+[gh-cube-openspec]: https://github.com/cube-js/cube/blob/master/packages/cubejs-api-gateway/openspec.yml
+[link-websocket]: https://en.wikipedia.org/wiki/WebSocket
+[ref-concurrency]: /product/configuration/concurrency
+[ref-data-warehouses]: /product/configuration/data-sources#data-warehouses
+[ref-traditional-databases]: /product/configuration/data-sources#transactional-databases
+[ref-pre-aggregations]: /product/caching/using-pre-aggregations
+[ref-javascript-sdk]: /product/apis-integrations/javascript-sdk
+[ref-recipe-real-time-data-fetch]: /product/apis-integrations/recipes/real-time-data-fetch

docs/pages/product/apis-integrations/rest-api/_meta.js

@@ -1,5 +1,4 @@
 module.exports = {
   "query-format": "Query format",
-  "real-time-data-fetch": "Real-Time data fetch",
   "reference": "Reference"
 }