Skip to content

add external data source public API for description #14438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 12, 2025
Merged

add external data source public API for description #14438

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
72fcd1c
add external data source public API for description
jepett0 Feb 11, 2025
810de8b
make properties a map + make content a map too
jepett0 Feb 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion ydb/public/api/grpc/ydb_table_v1.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ service TableService {
// Create new session. Implicit session creation is forbidden,
// so user must create new session before execute any query,
// otherwise BAD_SESSION status will be returned.
// Simultaneous execution of requests are forbiden.
// Simultaneous execution of requests are forbidden.
// Sessions are volatile, can be invalidated by server, for example in case
// of fatal errors. All requests with this session will fail with BAD_SESSION status.
// So, client must be able to handle BAD_SESSION status.
Expand Down Expand Up @@ -85,4 +85,10 @@ service TableService {

// Executes scan query with streaming result.
rpc StreamExecuteScanQuery(Table.ExecuteScanQueryRequest) returns (stream Table.ExecuteScanQueryPartialResponse);

// Returns information about a given external data source.
rpc DescribeExternalDataSource(Table.DescribeExternalDataSourceRequest) returns (Table.DescribeExternalDataSourceResponse);

// Returns information about a given external table.
rpc DescribeExternalTable(Table.DescribeExternalTableRequest) returns (Table.DescribeExternalTableResponse);
}
40 changes: 40 additions & 0 deletions ydb/public/api/protos/ydb_table.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1294,3 +1294,43 @@ message ExecuteScanQueryPartialResult {
// collects additional diagnostics about query compilation, including query plan and scheme
string query_full_diagnostics = 7;
}

// Returns information about an external data source with a given path.
message DescribeExternalDataSourceRequest {
Ydb.Operations.OperationParams operation_params = 1;
string path = 2;
}

message DescribeExternalDataSourceResponse {
// Holds DescribeExternalDataSourceResult in case of a successful call.
Ydb.Operations.Operation operation = 1;
}

message DescribeExternalDataSourceResult {
// Description of a generic scheme object.
Ydb.Scheme.Entry self = 1;
optional string source_type = 2;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

А какими могут быть значения этого поля? Это S3/PostgreSQL/MySQL/... ?

Copy link
Collaborator Author

@jepett0 jepett0 Feb 12, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Судя по доке:

В качестве источников данных можно использовать:

Но в коде main на данный момент написано следующее (этот код относится к External Table, смотри следующий коммент):

bool ValidateSourceType(const TString& sourceType, TString& errStr) {
    // Only object storage supported today
    if (sourceType != "ObjectStorage") {
        errStr = "Only ObjectStorage source type supported but got " + sourceType;
        return false;
    }
    return true;
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ага, кажется, я понимаю - этот код не совсем оттуда. Этот валидатор относится к CREATE EXTERNAL TABLE, и в этой команде действительно единственный поддерживаемый источник - это S3. А вот в CREATE EXTERNAL DATA SOURCE поддерживаемых источников гораздо больше.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Да, сорри, нашёл валидацию external data source. Тут список source type побольше, но полного я не нашёл (тут разобраны только некоторые опции):

 bool DataSourceMustHaveDataBaseName(const TProtoStringType& sourceType) const {
    return IsIn({"Greenplum", "PostgreSQL", "MySQL", "MsSQLServer", "ClickHouse"}, sourceType);
}

optional string location = 3;
map<string, string> properties = 4;
}

// Returns information about an external table with a given path.
message DescribeExternalTableRequest {
Ydb.Operations.OperationParams operation_params = 1;
string path = 2;
}

message DescribeExternalTableResponse {
// Holds DescribeExternalTableResult in case of a successful call.
Ydb.Operations.Operation operation = 1;
}

message DescribeExternalTableResult {
// Description of a generic scheme object.
Ydb.Scheme.Entry self = 1;
optional string source_type = 2;
optional string data_source_path = 3;
optional string location = 4;
repeated ColumnMeta columns = 5;
map<string, string> content = 6;
}
Loading