license and readme

ZhiHanZ · ZhiHanZ · commit 968c6644ded4 · 2022-12-09T17:28:07.000+08:00
diff --git a/src/binaries/README.md b/src/binaries/README.md
@@ -6,3 +6,4 @@ Contains all databend binaries:
 - [`metabench`](./metabench/) -> `databend-metabench`, run meta benchmark.
 - [`metactl`](./metactl/) -> `databend-metactl`, dump data in json from a sled db.
 - [`query`](./query/) -> `databend-query`, the query service binary of Databend.
+- [`opensharing`](./opensharing/) -> `open-sharing`, tenant level sharing endpoint used in stateful test
diff --git a/src/query/sharing-endpoint/README.md b/src/query/sharing-endpoint/README.md
@@ -0,0 +1,81 @@
+# Open Sharing
+Open Sharing is a cheap and secure data sharing protocol for databend query on multi-cloud environments.
+
+## Features
+* **Cheap**: Open Sharing allow data sharing via simple RESTful API sharing protocol, which is cheap and easy to understand.
+* **Secure**: Open Sharing protocol would verify allow incoming requesters identity and access permission and provide audit log.
+* **Multi-cloud**: Open Sharing is designed to work with different cloud platforms, including AWS, Azure, GCP, etc.
+* **Open source**: Open Sharing is an open source project
+
+## Protocol
+Databend Sharing protocol is a RESTful protocol, which would understand database and table semantics for data sharing.
+and provide short lived presigned url for data under the table.
+
+For detailed descriptions, please take a look at
+- [`protocol`](./protocol.md) -> provides detailed dscriptions on sharing protocol api
+
+## How to build?
+
+To build open-sharing for stateful tests, run the following command
+```bash
+cargo build --bin open-sharing
+```
+
+## How to use it?
+
+### Setup the sharing endpoint
+
+Please ensure that the sharing endpoint has read and list access for the bucket being shared.
+
+configure the sharing endpoint
+```bash
+export STORAGE_TYPE=s3
+export STORAGE_S3_REGION=<the shared bucket region>
+export STORAGE_S3_BUCKET=<the shared bucket name>
+export STORAGE_S3_ACCESS_KEY_ID=<the shared bucket access key id>
+export STORAGE_S3_SECRET_ACCESS_KEY=<the shared bucket secret access key>
+export STORAGE_S3_ROOT=<the shared bucket root path>
+export TENANT_ID=<the tenant id which shares the table>
+./open-sharing
+```
+
+### Setup the sharing endpoint address for databend query
+
+add `share_endpoint_address` field on your databend query config file
+
+```toml
+# Usage:
+# databend-query -c databend_query_config_spec.toml
+[query]
+...
+share_endpoint_address = "127.0.0.1:33003" # receive shared information from open sharing
+...
+```
+
+### How to share a table?
+
+For the tenant who wants to share the table `tabl1` from database `db1` to tenant `vendor`
+
+```sql
+CREATE SHARE myshare;
+GRANT USAGE ON DATABASE db1 TO SHARE myshare;
+GRANT SELECT ON TABLE db1.table1 TO SHARE myshare;
+ALTER SHARE myshare ADD TENANTS = vendor;
+```
+
+From tenant `vendor` side, the table `db1.table1` would be visible and can be queried.
+
+```sql
+CREATE DATABASE db2 FROM SHARE myshare;
+SELECT * FROM db2.table1;
+```
+
+## How to contribute?
+
+For code changes feel free to open a PR and add necessary unit tests and integration tests.
+
+For **API** changes, please follow the following steps:
+1. provide a RFC to explain the reason why we need the additional api or why we need to change the existing api.
+2. update the protocol.md to reflect the changes.
+3. update the implementation to reflect the changes.
+
diff --git a/src/query/sharing-endpoint/protocol.md b/src/query/sharing-endpoint/protocol.md
@@ -0,0 +1,72 @@
+# Databend Sharing protocol
+
+The Databend Sharing Protocol is a protocol that allows tenants in a multi-tenant database system
+to securely share files with each other. 
+
+The protocol defines a set of API endpoints that can be used to manage file sharing and access control.
+
+## Get Presigned File
+### POST /tenant/{tenant_id}/{share_name}/table/{table_name}/presign
+This endpoint allows a tenant to create a presigned URL for data files in a specified table. 
+
+#### Request
+
+```bash
+POST /tenant/{tenant_id}/{share_name}/table/{table_name}/presign
+```
+
+**tenant_id** : the tenant id who shares the table
+**share_name** : the share name
+**table_name** : the shared table name
+
+#### Headers
+
+| Name | Value | Description | Required |
+| ---- | ----- | ----------- |----------|
+| Authorization | Bearer {token} | The token used to authenticate the request. | Yes      |
+| Content-Type | application/json | The content type of the request. | No       |
+
+#### Body
+
+```json
+[
+  {
+    "file_name": "file1.txt",
+    "method": "GET"
+  },
+  {
+    "file_name": "file2.txt",
+    "method": "HEAD"
+  }
+]
+```
+
+An array of objects representing the files for which presigned URLs should be created. 
+
+Each object should contain the following properties:
+* **file_name**: The name of the file.
+* **method**: The HTTP method that should be used for the presigned URL. method should be either **GET** or **HEAD**.
+
+#### Response
+```json
+[
+    {
+        "presigned_url": "https://s3.example.com/table1/file1.txt?AWSAccessKeyId=ABC123&Expires=1560993041&Signature=def456",
+        "headers": {},
+        "method": "GET",
+        "path": "/table1/file1.txt"
+    },
+    {
+        "presigned_url": "https://s3.example.com/table1/file2.txt?AWSAccessKeyId=ABC123&Expires=1560993041&Signature=ghi789",
+        "headers": {},
+        "method": "HEAD",
+        "path": "/table1/file2.txt"
+    }
+]
+
+```
+
+* **presigned_url**: The presigned URL that can be used to access the file.
+* **headers**: An object containing any additional headers that should be included in the request to the presigned URL.
+* **method**: The HTTP method that is allowed for the presigned URL.
+* **path**: The path of the file relative to the table.
diff --git a/src/query/sharing-endpoint/src/accessor.rs b/src/query/sharing-endpoint/src/accessor.rs
@@ -1,3 +1,17 @@
+// Copyright 2022 Datafuse Labs.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use std::sync::Arc;
 
 use common_base::base::Singleton;
diff --git a/src/query/sharing-endpoint/src/handlers.rs b/src/query/sharing-endpoint/src/handlers.rs
@@ -1,3 +1,17 @@
+// Copyright 2022 Datafuse Labs.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use models::Credentials;
 use models::RequestFile;
 use poem::error::BadRequest;
diff --git a/src/query/sharing-endpoint/src/middlewares.rs b/src/query/sharing-endpoint/src/middlewares.rs
@@ -1,3 +1,17 @@
+// Copyright 2022 Datafuse Labs.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use poem::async_trait;
 use poem::Endpoint;
 use poem::IntoResponse;
diff --git a/src/query/sharing-endpoint/src/models.rs b/src/query/sharing-endpoint/src/models.rs
@@ -1,4 +1,4 @@
-// Copyright 2021 Datafuse Labs.
+// Copyright 2022 Datafuse Labs.
 //
 // Licensed under the Apache License, Version 2.0 (the "License");
 // you may not use this file except in compliance with the License.
diff --git a/src/query/sharing-endpoint/src/services.rs b/src/query/sharing-endpoint/src/services.rs
@@ -1,3 +1,17 @@
+// Copyright 2022 Datafuse Labs.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use std::cell::UnsafeCell;
 use std::sync::Arc;
 
diff --git a/src/query/sharing-endpoint/tests/it/accessor.rs b/src/query/sharing-endpoint/tests/it/accessor.rs
@@ -1,3 +1,17 @@
+// Copyright 2022 Datafuse Labs.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use common_base::base::tokio;
 use common_exception::Result;
 use sharing_endpoint::accessor::truncate_root;
diff --git a/src/query/sharing-endpoint/tests/it/models.rs b/src/query/sharing-endpoint/tests/it/models.rs
@@ -1,3 +1,17 @@
+// Copyright 2022 Datafuse Labs.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
 use std::collections::HashMap;
 
 use common_base::base::tokio;

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-// Copyright 2021 Datafuse Labs.`
	`1`	`+// Copyright 2022 Datafuse Labs.`
`2`	`2`	`//`
`3`	`3`	`// Licensed under the Apache License, Version 2.0 (the "License");`
`4`	`4`	`// you may not use this file except in compliance with the License.`