External data source: convert the private protobuf to the public one on describe RPC

[jepett0]

Convert the private protobuf description of External Data Source and External Table scheme objects to the public ones in the corresponding RPC implementing actors.

The conversion of the External Data Source protobuf is mostly the reversal of the conversion from the parser's map of features to the private protobuf executed here.

The conversion of the content of External Tables is copied from here.

Example

Create an external data source and an external table:

CREATE EXTERNAL DATA SOURCE external_data_source WITH (
    SOURCE_TYPE="ObjectStorage",
    LOCATION="192.168.1.1:8123",
    AUTH_METHOD="NONE"
);

CREATE EXTERNAL TABLE external_table (
    key Utf8 NOT NULL,
    value Utf8 NOT NULL
) WITH (
    DATA_SOURCE="external_data_source",
    LOCATION="folder",
    FORMAT="csv_with_names",
    COMPRESSION="gzip"
);

Now let's check the output of the ydb scheme describe command:

ydb scheme describe external_data_source --format proto-json-base64

<external-data-source> external_data_source
{"self":{"name":"external_data_source","owner":"root@builtin","type":"EXTERNAL_DATA_SOURCE","effective_permissions":[{"subject":"USERS","permission_names":["ydb.database.connect"]},{"subject":"METADATA-READERS","permission_names":["ydb.generic.list"]},{"subject":"DATA-READERS","permission_names":["ydb.granular.select_row"]},{"subject":"DATA-WRITERS","permission_names":["ydb.tables.modify"]},{"subject":"DDL-ADMINS","permission_names":["ydb.granular.create_directory","ydb.granular.write_attributes","ydb.granular.create_table","ydb.granular.remove_schema","ydb.granular.create_queue","ydb.granular.alter_schema"]},{"subject":"ACCESS-ADMINS","permission_names":["ydb.access.grant"]},{"subject":"DATABASE-ADMINS","permission_names":["ydb.generic.manage"]}],"created_at":{"plan_step":"1739462700490","tx_id":"281474976720659"}},"source_type":"ObjectStorage","location":"192.168.1.1:8123","properties":{"AUTH_METHOD":"NONE"}}

same, but without the description of "self" and formatted

{
  "source_type": "ObjectStorage",
  "location": "192.168.1.1:8123",
  "properties": {
    "AUTH_METHOD": "NONE"
  }
}

ydb scheme describe external_table --format proto-json-base64

<external-table> external_table
{"self":{"name":"external_table","owner":"root@builtin","type":"EXTERNAL_TABLE","effective_permissions":[{"subject":"USERS","permission_names":["ydb.database.connect"]},{"subject":"METADATA-READERS","permission_names":["ydb.generic.list"]},{"subject":"DATA-READERS","permission_names":["ydb.granular.select_row"]},{"subject":"DATA-WRITERS","permission_names":["ydb.tables.modify"]},{"subject":"DDL-ADMINS","permission_names":["ydb.granular.create_directory","ydb.granular.write_attributes","ydb.granular.create_table","ydb.granular.remove_schema","ydb.granular.create_queue","ydb.granular.alter_schema"]},{"subject":"ACCESS-ADMINS","permission_names":["ydb.access.grant"]},{"subject":"DATABASE-ADMINS","permission_names":["ydb.generic.manage"]}],"created_at":{"plan_step":"1739462700530","tx_id":"281474976720660"}},"source_type":"ObjectStorage","data_source_path":"/Root/dedicated/external_data_source","location":"folder","columns":[{"name":"key","type":{"type_id":"UTF8"}},{"name":"value","type":{"type_id":"UTF8"}}],"content":{"format":"[\"csv_with_names\"]","compression":"[\"gzip\"]"}}

same, but without the description of "self" and formatted

{
  "source_type": "ObjectStorage",
  "data_source_path": "/Root/dedicated/external_data_source",
  "location": "folder",
  "columns": [
    {
      "name": "key",
      "type": {
        "type_id": "UTF8"
      }
    },
    {
      "name": "value",
      "type": {
        "type_id": "UTF8"
      }
    }
  ],
  "content": {
    "format": "[\"csv_with_names\"]",
    "compression": "[\"gzip\"]"
  }
}

Changelog entry

Changelog category

• Not for changelog (changelog entry is not required)

Additional information

• external data sources: describe #14436

[dorooleg]

lgtm

[github-actions]

⚪ 2025-02-13 18:25:00 UTC Pre-commit check linux-x86_64-release-asan for 4bdde91 has started.
⚪ 2025-02-13 18:25:29 UTC Artifacts will be uploaded here
⚪ 2025-02-13 18:28:46 UTC ya make is running...
🟡 2025-02-13 20:00:40 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
11776	11696	0	22	21	37

⚪ 2025-02-13 20:01:46 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-13 20:13:54 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
120 (only retried tests)	80	0	0	8	32

🟢 2025-02-13 20:14:01 UTC Build successful.
🟡 2025-02-13 20:14:25 UTC ydbd size 3.6 GiB changed* by +328.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: 09744cf	merge: 4bdde91	diff	diff %
ydbd size	3 881 949 272 Bytes	3 882 285 960 Bytes	+328.8 KiB	+0.009%
ydbd stripped size	1 359 322 672 Bytes	1 359 348 880 Bytes	+25.6 KiB	+0.002%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

[github-actions]

⚪ 2025-02-13 18:34:25 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4bdde91 has started.
⚪ 2025-02-13 18:34:49 UTC Artifacts will be uploaded here
⚪ 2025-02-13 18:37:44 UTC ya make is running...
🟡 2025-02-13 19:28:40 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
19060	17710	0	4	1215	131

⚪ 2025-02-13 19:30:58 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-02-13 19:43:06 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED?
207 (only retried tests)	78	0	0	3	126

🟢 2025-02-13 19:43:13 UTC Build successful.
🟡 2025-02-13 19:43:32 UTC ydbd size 2.1 GiB changed* by +543.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: 68fb393	merge: 4bdde91	diff	diff %
ydbd size	2 231 793 880 Bytes	2 232 350 192 Bytes	+543.3 KiB	+0.025%
ydbd stripped size	473 690 616 Bytes	473 740 312 Bytes	+48.5 KiB	+0.010%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}