-
Notifications
You must be signed in to change notification settings - Fork 702
add external data source public API for description #14438
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 1 commit
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1294,3 +1294,92 @@ message ExecuteScanQueryPartialResult { | |
// collects additional diagnostics about query compilation, including query plan and scheme | ||
string query_full_diagnostics = 7; | ||
} | ||
|
||
// Returns information about an external data source with a given path. | ||
message DescribeExternalDataSourceRequest { | ||
Ydb.Operations.OperationParams operation_params = 1; | ||
string path = 2; | ||
} | ||
|
||
message DescribeExternalDataSourceResponse { | ||
// Holds DescribeExternalDataSourceResult in case of a successful call. | ||
Ydb.Operations.Operation operation = 1; | ||
} | ||
|
||
message ExternalAuthorization { | ||
|
||
// Access without authorization. | ||
message None { | ||
} | ||
|
||
message Basic { | ||
optional string login = 1; | ||
optional string password_secret_name = 2; | ||
} | ||
|
||
message Token { | ||
optional string token_secret_name = 1; | ||
} | ||
|
||
message Aws { | ||
optional string access_key_secret_name = 1; | ||
optional string secret_key_secret_name = 2; | ||
optional string aws_region = 3; | ||
} | ||
|
||
message ServiceAccount { | ||
optional string id = 1; | ||
optional string secret_name = 2; | ||
} | ||
|
||
message MdbBasic { | ||
optional string service_account_id = 1; | ||
optional string service_account_secret_name = 2; | ||
optional string login = 3; | ||
optional string password_secret_name = 4; | ||
} | ||
|
||
oneof identity { | ||
None none = 1; | ||
Basic basic = 2; | ||
Token token = 3; | ||
Aws aws = 4; | ||
ServiceAccount service_account = 5; | ||
MdbBasic mdb_basic = 6; | ||
} | ||
} | ||
|
||
message ExternalDataSourceProperties { | ||
map<string, string> properties = 1; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Почему это упаковано в отдельное сообщение? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Когда это было на уровне ModifyScheme SS, то была идея это сделать отдельным protobuf чтобы можно было какие-то типизированные настройки добавлять. Но пока что не пригодилось) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Выложил на верхний уровень DescribeExternalDataSourceResult'а и объединил с ExternalAuthorization, которые тоже развернул в map<string, string>. Подробнее в этом комменте |
||
} | ||
|
||
message DescribeExternalDataSourceResult { | ||
// Description of a generic scheme object. | ||
Ydb.Scheme.Entry self = 1; | ||
optional string source_type = 2; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. А какими могут быть значения этого поля? Это There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Судя по доке: В качестве источников данных можно использовать: Но в коде main на данный момент написано следующее (этот код относится к External Table, смотри следующий коммент): bool ValidateSourceType(const TString& sourceType, TString& errStr) {
// Only object storage supported today
if (sourceType != "ObjectStorage") {
errStr = "Only ObjectStorage source type supported but got " + sourceType;
return false;
}
return true;
} There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ага, кажется, я понимаю - этот код не совсем оттуда. Этот валидатор относится к There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Да, сорри, нашёл валидацию external data source. Тут список source type побольше, но полного я не нашёл (тут разобраны только некоторые опции): bool DataSourceMustHaveDataBaseName(const TProtoStringType& sourceType) const {
return IsIn({"Greenplum", "PostgreSQL", "MySQL", "MsSQLServer", "ClickHouse"}, sourceType);
} |
||
optional string location = 3; | ||
optional string installation = 4; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Installation сейчас не используется, предлагаю когда имплементация появится, то добавить это поле There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. убрал |
||
optional ExternalAuthorization authorization = 5; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Возможно authorization стоит развернуть и переместить в There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Развернул Authorization в общий map<string, string> properties. Кажется, такое описание лучше ложится на:
Иначе говоря, для создания external data source через публичный API (на данных момент это исключительно YQL) нужно собрать такую вот map of features. Поэтому естественно здесь, в describe, отдавать именно её. Дополнительное усложнение в виде выделения ExternalAuthorization кажется избыточным, потому что такой структуры нет в |
||
optional ExternalDataSourceProperties properties = 6; | ||
} | ||
|
||
// Returns information about an external table with a given path. | ||
message DescribeExternalTableRequest { | ||
Ydb.Operations.OperationParams operation_params = 1; | ||
string path = 2; | ||
} | ||
|
||
message DescribeExternalTableResponse { | ||
// Holds DescribeExternalTableResult in case of a successful call. | ||
Ydb.Operations.Operation operation = 1; | ||
} | ||
|
||
message DescribeExternalTableResult { | ||
// Description of a generic scheme object. | ||
Ydb.Scheme.Entry self = 1; | ||
optional string source_type = 2; | ||
optional string data_source_path = 3; | ||
optional string location = 4; | ||
repeated ColumnMeta columns = 5; | ||
optional bytes content = 6; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Content лучше развернуть в какой нибудь json/map<string, string> или же any с конкретными protobuf There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Развернул в map<string, string>. Для полноты:
|
||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
У нас это нигде явно не зафиксировано, но мы называем такую аутентификацию "статической": https://ydb.tech/docs/ru/recipes/ydb-sdk/auth-static
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Похоже мы разъехались в этом месте, тут уже упоминается как basic https://ydb.tech/docs/ru/concepts/federated_query/clickhouse :(
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Названия в данном месте не обязаны совпадать. Это различия между кредами к ydb и кредами к внешней системе. Статические креды-это просто некий класс кредов, можете в инете почитать
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Засунул авторизацию в общее поле map<string, string> properties. Теперь уже не важно, Static или Basic. Подробнее в этом комменте