Add tt tdg2 import and export (#4171)

Resolves #4145

Co-authored-by: Georgy Moiseev <moiseev.georgii@gmail.com>
Co-authored-by: Andrey Aksenov <38073144+andreyaksenov@users.noreply.github.com>

Author: 3 people

doc/reference/tooling/tt_cli/commands.rst

@@ -70,6 +70,8 @@ help for the given command.
             -   Get the current status of a Tarantool instance
         *   -   :doc:`stop <stop>`
             -   Stop a Tarantool instance
+        *   -   :doc:`tdg2 <tdg2>`
+            -   Interact with `Tarantool Data Grid 2 <https://www.tarantool.io/ru/tdg/latest/>`_ clusters
         *   -   :doc:`uninstall <uninstall>`
             -   Uninstall Tarantool or ``tt``
         *   -   :doc:`version <version>`
@@ -107,5 +109,6 @@ help for the given command.
     start <start>
     status <status>
     stop <stop>
+    tdg2 <tdg2>
     uninstall <uninstall>
     version <version>

doc/reference/tooling/tt_cli/export.rst

@@ -11,12 +11,17 @@ Exporting data
 
 ..  code-block:: console
 
-    $ tt [crud] export URI SPACE:FILE ... [EXPORT_OPTION ...]
+    $ tt [crud|tdg2] export URI SPACE:FILE ... [EXPORT_OPTION ...]
 
-``tt [crud] export`` exports a space's data to a file.
-The ``crud`` command is optional and can be used to export a cluster's data by using the `CRUD <https://github.com/tarantool/crud>`_ module. Without ``crud``, data is exported using the :ref:`box.space <box_space>` API.
+``tt [crud|tdg2] export`` exports a space's data to a file. Three export commands
+cover the following cases:
 
-``tt [crud] export`` takes the following arguments:
+*   ``tt export`` exports data from a replica set using the :ref:`box.space <box_space>` API.
+*   ``tt crud export`` exports data from a sharded cluster through a router using the `CRUD <https://github.com/tarantool/crud>`_ module.
+*   ``tt tdg2 export`` exports data from a `Tarantool Data Grid 2 <https://www.tarantool.io/ru/tdg/latest/>`_ cluster
+    through its `connector <https://www.tarantool.io/ru/tdg/latest/architecture/#connector>`_ using `TDG2 Repository API <https://www.tarantool.io/en/tdg/latest/reference/sandbox/repository-api/#repository-api>`_.
+
+``tt [crud|tdg2] export`` takes the following arguments:
 
 *   ``URI``: The URI of a router instance if ``crud`` is used. Otherwise, it should specify the URI of a storage.
 *   ``FILE``: The name of a file for storing exported data.
@@ -26,6 +31,16 @@ The ``crud`` command is optional and can be used to export a cluster's data by u
 
     :ref:`Read access <authentication-owners_privileges>` to the space is required to export its data.
 
+.. _tt-export-output-format:
+
+Output format
+-------------
+
+``tt export`` exports data in the following formats:
+
+*   ``tt export`` and ``tt crud export``: CSV
+*   ``tt tdg2 export``: JSON lines
+
 .. _tt-export-limitations:
 
 Limitations
@@ -66,7 +81,7 @@ If a tuple contains a ``null`` value, for example, ``[1, 477, 'Andrew', null, 38
 Exporting headers
 -----------------
 
-To export data with a space's field names in the first row, use the ``--header`` option:
+To export data with a space's field names in the first row of the CSV file, use the ``--header`` option:
 
 .. code-block:: console
 
@@ -89,15 +104,69 @@ In this case, field values start from the second row, for example:
 Exporting compound data
 -----------------------
 
-By default, ``tt`` exports empty values for fields containing compound data such as arrays or maps.
+In the CSV format, ``tt`` exports empty values by default for fields containing compound data such as arrays or maps.
 To export compound values in a specific format, use the ``--compound-value-format`` option.
-For example, the command below exports compound values serialized in JSON:
+For example, the command below exports compound values to CSV serialized in JSON:
 
 .. code-block:: console
 
     $ tt crud export localhost:3301 customers:customers.csv  \
                      --compound-value-format json
 
+.. _tt-export-tdg2:
+
+Exporting from Tarantool Data Grid 2
+------------------------------------
+
+.. note::
+
+    In the TDG2 data model, a **type** represents a Tarantool space, and an **object**
+    of a type represents a tuple in the type's underlying space.
+
+The command below exports data of the ``customers`` type from a TDG2 cluster to
+the ``customers.jsonl`` file:
+
+.. code-block:: console
+
+    $ tt tdg2 export localhost:3301 customers:customers.jsonl
+
+If the ``customers`` type has four fields (``id``, ``firstname``, ``lastname``, and ``age``), the file with exported data might look like this:
+
+.. code-block:: json
+
+    {"age":30,"first_name":"Samantha","id":1,"second_name":"Carter"}
+    {"age":41,"first_name":"Fay","id":2,"second_name":"Rivers"}
+    {"age":74,"first_name":"Milo","id":4,"second_name":"Walters"}
+
+If an object contains a ``null`` value in a field, this field skipped:
+
+.. code-block:: json
+
+    {"age":13,"first_name":"Zachariah","id":3}
+
+Object fields that contain maps with non-string keys are converted to maps with string keys.
+
+TDG2 sets a limit on the number of objects transferred from each storage during a query execution
+(the `hard-limits.returned <https://www.tarantool.io/en/tdg/latest/reference/config/config_logic/#hard-limits>`_
+TDG2 configuration parameter). If an export batch size (``--batch-size`` parameter)
+is greater than this limit, it is possible that more than ``hard-limits.returned`` objects
+will be requested from one storage and export will fail.
+To make sure that ``hard-limits.returned`` is never exceeded during an export operation,
+set the export batch size less or equal to this limit.
+
+For example, if your TDG2 cluster has a 1000 objects ``hard-limits.returned`` limit:
+
+.. code-block:: yaml
+
+    # tdg2 config.yaml
+    # ...
+    hard-limits.returned: 1000
+
+Set the ``tt tdg2 export`` batch size less or equal to 1000:
+
+.. code-block:: console
+
+    $ tt tdg2 export localhost:3301 customers:customers.jsonl --batch-size=1000
 
 .. _tt-export-options:
 
@@ -118,10 +187,20 @@ Options
 
 ..  option:: --batch-size INT
 
-    The number of tuples to transfer per request (the default is ``10000``).
+    The number of tuples to transfer per request. The default is:
+
+        *   ``10000`` for ``tt export`` and ``tt crud export``.
+        *   ``100`` for ``tt tdg2 export``.
+
+    .. important::
+
+        When using ``tt tdg2 export``, make sure that the batch size does not exceed
+        the ``hard-limits.returned`` TDG2 parameter value set on the cluster.
 
 ..  option:: --compound-value-format STRING
 
+    **Applicable to:** ``tt export``, ``tt crud export``
+
     A format used to export compound values like arrays or maps.
     By default, ``tt`` exports empty values for fields containing such values.
 
@@ -131,6 +210,8 @@ Options
 
 ..  option:: --header
 
+    **Applicable to:** ``tt export``, ``tt crud export``
+
     Add field names in the first row.
 
     See also: :ref:`Exporting headers <tt-export-header>`.
@@ -141,6 +222,8 @@ Options
 
 ..  option:: --readview
 
+    **Applicable to:** ``tt export``, ``tt crud export``
+
     Export data using a :ref:`read view <read_views>`.
 
 ..  option:: --username STRING

doc/reference/tooling/tt_cli/import.rst

@@ -11,14 +11,19 @@ Importing data
 
 .. code-block:: console
 
-    $ tt [crud] import URI FILE:SPACE [IMPORT_OPTION ...]
+    $ tt [crud|tdg2] import URI FILE:SPACE [IMPORT_OPTION ...]
     # or
-    $ tt [crud] import URI :SPACE < FILE [IMPORT_OPTION ...]
+    $ tt [crud|tdg2] import URI :SPACE < FILE [IMPORT_OPTION ...]
 
-``tt [crud] import`` imports data from a file to a space.
-The ``crud`` command is optional and can be used to import data to a cluster by using the `CRUD <https://github.com/tarantool/crud>`_ module. Without ``crud``, data is imported using the :ref:`box.space <box_space>` API.
+``tt [crud|tdg] import`` imports data from a file to a space. Three import commands
+cover the following cases:
 
-This command takes the following arguments:
+*   ``tt import`` imports data into a replica set through its master instance using the :ref:`box.space <box_space>` API.
+*   ``tt crud import`` imports data into a sharded cluster through a router using the `CRUD <https://github.com/tarantool/crud>`_ module.
+*   ``tt tdg2 import`` imports data into a `Tarantool Data Grid 2 <https://www.tarantool.io/ru/tdg/latest/>`_ cluster
+    through its router using the ``repository.put`` function of the `TDG2 Repository API <https://www.tarantool.io/en/tdg/latest/reference/sandbox/repository-api/#repository-api>`_.
+
+``tt [crud|tdg2] import`` takes the following arguments:
 
 *   ``URI``: The URI of a router instance if ``crud`` is used. Otherwise, it should specify the URI of a storage.
 *   ``FILE``: The name of a file containing data to be imported.
@@ -28,6 +33,15 @@ This command takes the following arguments:
 
     :ref:`Write access <authentication-owners_privileges>` to the space and `execute` access to `universe` are required to import data.
 
+.. _tt-import-format:
+
+Input file format
+-----------------
+
+``tt import`` imports data from the following formats:
+
+*   ``tt import`` and ``tt crud import``: CSV
+*   ``tt tdg2 import``: JSON lines
 
 .. _tt-import-limitations:
 
@@ -134,14 +148,77 @@ To skip rows whose data cannot be parsed correctly, use the ``--on-error`` optio
     $ tt crud import localhost:3301 customers.csv:customers \
                      --on-error skip
 
+.. _tt-import-tdg2:
+
+Importing into Tarantool Data Grid 2
+------------------------------------
+
+.. note::
+
+    In the TDG2 data model, a **type** represents a Tarantool space, and an **object**
+    of a type represents a tuple in the type's underlying space.
+
+The command below imports objects of the ``customers`` type into a TDG2 cluster.
+The objects are described in the ``customers.jsonl`` file.
+
+.. code-block:: console
+
+    $ tt tdg2 import localhost:3301 customers.jsonl:customers
+
+The input file can look like this:
+
+.. code-block:: json
+
+    {"age":30,"first_name":"Samantha","id":1,"second_name":"Carter"}
+    {"age":41,"first_name":"Fay","id":2,"second_name":"Rivers"}
+    {"age":74,"first_name":"Milo","id":4,"second_name":"Walters"}
+
+.. note::
+
+    Since JSON describes objects in maps with string keys, there is no way to
+    import a field value that is a map with a non-string key.
+
+In case of an error during TDG2 import, ``tt tdg2 import`` rolls back the changes made
+*within the current batch* on the *storage where the error has happened* (per-storage rollback)
+and reports an error. On other storages, objects from the same batch can be successfully
+imported. So, the rollback process of ``tt tdg2 import``
+is the same as the one of ``tt crud import`` with the ``--rollback-on-error`` option.
+
+Since object batches can be imported partially (per-storage rollback), the absence
+of error matching complicates the debugging in case of errors. To minimize this
+effect, the default batch size (``--batch-size``) for ``tt tdg2 import`` is 1.
+This makes the debugging straightforward: you always know which object caused the error.
+On the other hand, this decreases the performance in comparison to import in larger batches.
+
+If you increase the batch size, ``tt`` informs you about the possible issues and
+asks for an explicit confirmation to proceed.
+To automatically confirm a batch import operation, add the ``--force`` option:
+
+.. code-block:: console
+
+    $ tt tdg2 import localhost:3301 customers.jsonl:customers \
+                     --batch-size=100 \
+                     --force
+
 
 .. _tt-import-options:
 
 Options
 -------
 
+..  option:: --batch-size INT
+
+    **Applicable to:** ``tt crud import``, ``tt tdg2 import``
+
+    The number of tuples to transfer per request. The default is:
+
+        *   ``100`` for ``tt crud import``.
+        *   ``1`` for ``tt tdg2 import``. See :ref:`tt-import-tdg2` for details.
+
 ..  option:: --dec-sep STRING
 
+    **Applicable to:** ``tt import``, ``tt crud import``
+
     The string of symbols that defines decimal separators for numeric data (the default is ``.,``).
 
     .. NOTE::
@@ -150,6 +227,8 @@ Options
 
 ..  option:: --delimiter STRING
 
+    **Applicable to:** ``tt import``, ``tt crud import``
+
     A symbol that defines a field value delimiter.
     For CSV, the default delimiter is a comma (``,``).
     To use a tab character as a delimiter, set this value as ``tab``:
@@ -169,6 +248,12 @@ Options
 
     See also: :ref:`Handling parsing errors <tt-import-parsing-error>`.
 
+..  option:: --force
+
+    **Applicable to:** ``tt tdg2 import``
+
+    Automatically confirm importing into TDG2 with ``--batch-size`` greater than one.
+
 ..  option:: --format STRING
 
     A format of input data.
@@ -177,6 +262,8 @@ Options
 
 ..  option:: --header
 
+    **Applicable to:** ``tt import``, ``tt crud import``
+
     Process the first line as a header containing field names.
     In this case, field values start from the second line.
 
@@ -189,12 +276,16 @@ Options
 
 ..  option:: --match STRING
 
+    **Applicable to:** ``tt import``, ``tt crud import``
+
     Configure matching between field names in the input file and the target space.
 
     See also: :ref:`Matching of input and space fields <tt-import-match-fields>`.
 
 ..  option:: --null STRING
 
+    **Applicable to:** ``tt import``, ``tt crud import``
+
     A value to be interpreted as ``null`` when importing data.
     By default, an empty value is interpreted as ``null``.
     For example, a tuple imported from the following row ...
@@ -253,6 +344,8 @@ Options
 
 ..  option:: --quote STRING
 
+    **Applicable to:** ``tt import``, ``tt crud import``
+
     A symbol that defines a quote.
     For CSV, double quotes are used by default (``"``).
     The double symbol of this option acts as the escaping symbol within input data.
@@ -264,6 +357,8 @@ Options
 
 ..  option:: --th-sep STRING
 
+    **Applicable to:** ``tt import``, ``tt crud import``
+
     The string of symbols that define thousand separators for numeric data.
     The default value includes a space and a backtick `````.
     This means that ``1 000 000`` and ``1`000`000`` are both imported as ``1000000``.
@@ -278,6 +373,11 @@ Options
 
 ..  option:: --rollback-on-error
 
-    Applicable only when ``crud`` is used.
+    **Applicable to:** ``tt crud import``
+
+    Specify whether any operation failed on a storage leads to rollback of a batch
+    import on this storage.
+
+    .. note::
 
-    Specify whether any operation failed on a router leads to rollback on a storage where the operation is failed.
+        ``tt tdg2 import`` always works as if ``--rollback-on-error`` is ``true``.

doc/reference/tooling/tt_cli/tdg2.rst

@@ -0,0 +1,19 @@
+.. _tt-tdg2:
+
+Interacting with the Tarantool Data Grid 2
+==========================================
+
+..  admonition:: Enterprise Edition
+    :class: fact
+
+    This command is supported by the `Enterprise Edition <https://www.tarantool.io/compare/>`_ only.
+
+..  code-block:: console
+
+    $ tt tdg2 COMMAND [COMMAND_OPTION ...]
+
+``tt tdg2`` enables the interaction with `Tarantool Data Grid 2 <https://www.tarantool.io/ru/tdg/latest/>`_ clusters.
+``COMMAND`` is one of the following:
+
+*   ``export``: export a TDG2 cluster's data to a file. Learn more at :ref:`Exporting data <tt-export>`.
+*   ``import``: import data to a TDG2 cluster from a file. Learn more at :ref:`Importing data <tt-import>`.