From d68b642dbb39f12e12dfcbe96ea243a375f068a5 Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 06:36:48 -1000 Subject: [PATCH 01/11] Add docs for Private Link using VPC Resource --- .../clickpipes/aws-privatelink.md | 79 ++++++++++++++++--- 1 file changed, 69 insertions(+), 10 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index d9fefe23dc2..e91ac37d070 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -32,26 +32,85 @@ ClickPipes reverse private endpoint can be configured with one of the following - [MSK multi-VPC connectivity for MSK ClickPipe](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) - [VPC endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html) -Follow the links above for detailed instructions on how to set up the respective AWS PrivateLink shares. - ### VPC resource {#vpc-resource} -Your VPC resources can be accessed in ClickPipes using PrivateLink. +Your VPC resources can be accessed in ClickPipes using PrivateLink and [AWS VPC Lattice](https://docs.aws.amazon.com/vpc-lattice/latest/ug/what-is-vpc-lattice.html). This approach doesn't require setting up a load balancer in front of your data source. + Resource configuration can be targeted with a specific host or RDS cluster ARN. Cross-region is not supported. It's the preferred choice for Postgres CDC ingesting data from an RDS cluster. -See a [getting started](https://docs.aws.amazon.com/vpc/latest/privatelink/resource-configuration.html) guide for more details. +To set up PrivateLinke with VPC resource: +1. Create a Resource-Gateway +2. Create a Resource-Configuration +3. Share the Resource-Configuration and Resource-Owner Account ID with ClickPipes -:::info -VPC resource needs to be shared with a ClickPipes account. Add `072088201116` to the allowed principals to your resource share configuration. -See AWS guide for [sharing resources](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) for more details. -::: +#### 1. Create a Resource-Gateway + +Resource-Gateway is the point that receives traffic for specified resources in your VPC. + +You can create a Resource-Gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command: + +```bash +aws vpc-lattice create-resource-gateway \ +--vpc-identifier \ +--subnet-ids \ +--security-group-ids \ +--name +``` + +Before you can proceed, wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command: + +```bash +aws vpc-lattice get-resource-gateway \ +--resource-gateway-identifier +``` + +#### 2. Create a VPC Resource-Configuration + +Resource-Configuration is associated with Resource-Gateway to make your resource accessible. + +You can create a Resource-Configuration from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-configuration.html) or with the following command: + +```bash +aws vpc-lattice create-resource-configuration \ +--resource-gateway-identifier \ +--type \ +--resource-configuration-definition \ +--name +``` + +The simpliest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is Single Resource-Configuration. This allows you to share an IP-address or a domain-name that is publicly resolvable. + +#### 3. Share the Resource Configuration with ClickHouse + +Sharing your resource requires the Resource-Configuration and Resource-Owner Account ID to be shared with ClickPipes, this is facilitated through the Resource Access Manager (RAM). + +You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116`: + +```bash +aws ram create-resource-share \ +--principals 072088201116 \ +--resource-arns \ +--name +``` + +Finally, to share your Resource-Owner Account ID with the ClickPipes team, you can raise a [support ticket](https://console.clickhouse.cloud/support) through the ClickHouse Console and provide your Resource Owner Account ID. + +You can find this account ID within the [AWS console](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-identifiers.html) or by running the following CLI command: + +```bash +aws sts get-caller-identity \ +--query Account \ +--output text +``` + +For more details on VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). ### MSK multi-VPC connectivity {#msk-multi-vpc} -The MSK multi-VPC is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster. +The [Multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster. Private DNS support is out of the box and does not require any additional configuration. Cross-region is not supported. @@ -67,7 +126,7 @@ Follow our [MSK setup guide for ClickPipes](/knowledgebase/aws-privatelink-setup ### VPC endpoint service {#vpc-endpoint-service} -VPC service is another approach to share your data source with ClickPipes. +[VPC endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html) is another approach to share your data source with ClickPipes. It requires setting up a NLB (Network Load Balancer) in front of your data source and configuring the VPC endpoint service to use the NLB. From 9e1796522bb5162d32de625aa2ca4d678d3dea4a Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 06:38:00 -1000 Subject: [PATCH 02/11] typo --- docs/integrations/data-ingestion/clickpipes/aws-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index e91ac37d070..ba24bc0b443 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -41,7 +41,7 @@ Cross-region is not supported. It's the preferred choice for Postgres CDC ingesting data from an RDS cluster. -To set up PrivateLinke with VPC resource: +To set up PrivateLink with VPC resource: 1. Create a Resource-Gateway 2. Create a Resource-Configuration 3. Share the Resource-Configuration and Resource-Owner Account ID with ClickPipes From 6ef914c62f85dd504f84d1aff88fd241634d2eb6 Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 07:00:05 -1000 Subject: [PATCH 03/11] fixes --- .../data-ingestion/clickpipes/aws-privatelink.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index ba24bc0b443..3c65b3b2752 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -42,11 +42,11 @@ Cross-region is not supported. It's the preferred choice for Postgres CDC ingesting data from an RDS cluster. To set up PrivateLink with VPC resource: -1. Create a Resource-Gateway -2. Create a Resource-Configuration -3. Share the Resource-Configuration and Resource-Owner Account ID with ClickPipes +1. Create a resource gateway +2. Create a resource configuration +3. Share the resource configuration and resource owner acount id with ClickPipes team -#### 1. Create a Resource-Gateway +#### 1. Create a Resource-Gateway {#create-resource-gateway} Resource-Gateway is the point that receives traffic for specified resources in your VPC. @@ -67,7 +67,7 @@ aws vpc-lattice get-resource-gateway \ --resource-gateway-identifier ``` -#### 2. Create a VPC Resource-Configuration +#### 2. Create a VPC Resource-Configuration {#create-resource-configuration} Resource-Configuration is associated with Resource-Gateway to make your resource accessible. @@ -81,9 +81,9 @@ aws vpc-lattice create-resource-configuration \ --name ``` -The simpliest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is Single Resource-Configuration. This allows you to share an IP-address or a domain-name that is publicly resolvable. +The simplest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is Single Resource-Configuration. This allows you to share an IP-address or a domain-name that is publicly resolvable. -#### 3. Share the Resource Configuration with ClickHouse +#### 3. Share the Resource Configuration with ClickHouse {#share-resource-configuration} Sharing your resource requires the Resource-Configuration and Resource-Owner Account ID to be shared with ClickPipes, this is facilitated through the Resource Access Manager (RAM). From 7872adc3388be8b4714631c427b9c3989306365c Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 07:02:05 -1000 Subject: [PATCH 04/11] spelling --- docs/integrations/data-ingestion/clickpipes/aws-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 3c65b3b2752..75c2721b1d4 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -44,7 +44,7 @@ It's the preferred choice for Postgres CDC ingesting data from an RDS cluster. To set up PrivateLink with VPC resource: 1. Create a resource gateway 2. Create a resource configuration -3. Share the resource configuration and resource owner acount id with ClickPipes team +3. Share the resource configuration and resource owner account id with ClickPipes team #### 1. Create a Resource-Gateway {#create-resource-gateway} From 2d1be582b27c3a7625beb0936cd53edf95ece51c Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 10:11:54 -1000 Subject: [PATCH 05/11] pr review --- .../clickpipes/aws-privatelink.md | 36 ++++++++++++------- 1 file changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 75c2721b1d4..24df2c21c37 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -44,7 +44,7 @@ It's the preferred choice for Postgres CDC ingesting data from an RDS cluster. To set up PrivateLink with VPC resource: 1. Create a resource gateway 2. Create a resource configuration -3. Share the resource configuration and resource owner account id with ClickPipes team +3. Create a resource share #### 1. Create a Resource-Gateway {#create-resource-gateway} @@ -60,6 +60,8 @@ aws vpc-lattice create-resource-gateway \ --name ``` +The output will contain a Resource-Gateway id, which you will need for the next step. + Before you can proceed, wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command: ```bash @@ -81,11 +83,22 @@ aws vpc-lattice create-resource-configuration \ --name ``` -The simplest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is Single Resource-Configuration. This allows you to share an IP-address or a domain-name that is publicly resolvable. +The simplest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is a single Resource-Configuration. You can configure with the ARN directly, or share an IP address or a domain name that is publicly resolvable. + +For example, to configure with the ARN of an RDS Cluster: +``` +aws vpc-lattice create-resource-configuration \ + --name my-rds-cluster-config \ + --type ARN \ + --resource-gateway-identifier rgw-0bba03f3d56060135 \ + --resource-configuration-definition 'arnResource={arn=arn:aws:rds:us-east-1:123456789012:cluster:my-rds-cluster}' +``` -#### 3. Share the Resource Configuration with ClickHouse {#share-resource-configuration} +The output will contain a Resource-Configuration ARN, which you will need for the next step. It will also contain a Resource-Configuration ID, which you will need to set up a ClickPipe connection with VPC resource. -Sharing your resource requires the Resource-Configuration and Resource-Owner Account ID to be shared with ClickPipes, this is facilitated through the Resource Access Manager (RAM). +#### 3. Create a Resource-Share {#create-resource-share} + +Sharing your resource requires a Resource-Share. This is facilitated through the Resource Access Manager (RAM). You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116`: @@ -96,17 +109,14 @@ aws ram create-resource-share \ --name ``` -Finally, to share your Resource-Owner Account ID with the ClickPipes team, you can raise a [support ticket](https://console.clickhouse.cloud/support) through the ClickHouse Console and provide your Resource Owner Account ID. - -You can find this account ID within the [AWS console](https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-identifiers.html) or by running the following CLI command: +The output will contain a Resource-Share ARN, which you will need to set up a ClickPipe connection with VPC resource. -```bash -aws sts get-caller-identity \ ---query Account \ ---output text -``` +You are ready to [create a ClickPipe with Reverse private endpoint](#creating-clickpipe) using VPC resource. You will need to: +- Set `VPC endpoint type` to `VPC Resource`. +- Set `Resource configuration ID` to the ID of the Resource-Configuration created in step 2. +- Set `Resource share ARN` to the ARN of the Resource-Share created in step 3. -For more details on VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). +For more details on PrivateLink with VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). ### MSK multi-VPC connectivity {#msk-multi-vpc} From a7546b0fa0865ea344beeff751b99d4d90d1cedb Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 10:14:04 -1000 Subject: [PATCH 06/11] more pr review --- docs/integrations/data-ingestion/clickpipes/aws-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 24df2c21c37..86ee200d6a1 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -62,7 +62,7 @@ aws vpc-lattice create-resource-gateway \ The output will contain a Resource-Gateway id, which you will need for the next step. -Before you can proceed, wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command: +Before you can proceed, you'll need to wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command: ```bash aws vpc-lattice get-resource-gateway \ From 9ee81613b18aea7375be0a15be1492b38f70b579 Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 10:15:38 -1000 Subject: [PATCH 07/11] ci --- docs/integrations/data-ingestion/clickpipes/aws-privatelink.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 86ee200d6a1..9494217e86c 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -86,7 +86,8 @@ aws vpc-lattice create-resource-configuration \ The simplest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is a single Resource-Configuration. You can configure with the ARN directly, or share an IP address or a domain name that is publicly resolvable. For example, to configure with the ARN of an RDS Cluster: -``` + +```bash aws vpc-lattice create-resource-configuration \ --name my-rds-cluster-config \ --type ARN \ From d343ca5a8f53d7ee4144188102218e8893daf388 Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 13:02:01 -1000 Subject: [PATCH 08/11] Update docs/integrations/data-ingestion/clickpipes/aws-privatelink.md Co-authored-by: Kunal Gupta <39487888+iamKunalGupta@users.noreply.github.com> --- .../data-ingestion/clickpipes/aws-privatelink.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 9494217e86c..de1d88f8932 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -54,10 +54,10 @@ You can create a Resource-Gateway from the [AWS console](https://docs.aws.amazon ```bash aws vpc-lattice create-resource-gateway \ ---vpc-identifier \ ---subnet-ids \ ---security-group-ids \ ---name + --vpc-identifier \ + --subnet-ids \ + --security-group-ids \ + --name ``` The output will contain a Resource-Gateway id, which you will need for the next step. From 7351b42c9593381e9ecc3d3fd0ff82cb5edb3eae Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 13:02:11 -1000 Subject: [PATCH 09/11] Update docs/integrations/data-ingestion/clickpipes/aws-privatelink.md Co-authored-by: Kunal Gupta <39487888+iamKunalGupta@users.noreply.github.com> --- .../data-ingestion/clickpipes/aws-privatelink.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index de1d88f8932..364fa71c416 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -77,10 +77,10 @@ You can create a Resource-Configuration from the [AWS console](https://docs.aws. ```bash aws vpc-lattice create-resource-configuration \ ---resource-gateway-identifier \ ---type \ ---resource-configuration-definition \ ---name + --resource-gateway-identifier \ + --type \ + --resource-configuration-definition \ + --name ``` The simplest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is a single Resource-Configuration. You can configure with the ARN directly, or share an IP address or a domain name that is publicly resolvable. From ef8cd8be5160d33d8114673502df547a0740e58f Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 13:02:18 -1000 Subject: [PATCH 10/11] Update docs/integrations/data-ingestion/clickpipes/aws-privatelink.md Co-authored-by: Kunal Gupta <39487888+iamKunalGupta@users.noreply.github.com> --- .../data-ingestion/clickpipes/aws-privatelink.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 364fa71c416..50b61cd5b14 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -105,9 +105,9 @@ You can put the Resource-Configuration into the Resource-Share through [AWS cons ```bash aws ram create-resource-share \ ---principals 072088201116 \ ---resource-arns \ ---name + --principals 072088201116 \ + --resource-arns \ + --name ``` The output will contain a Resource-Share ARN, which you will need to set up a ClickPipe connection with VPC resource. From ac1942187676bdf9b1ecac86b624933eda7b980e Mon Sep 17 00:00:00 2001 From: Joy Gao <17896160+jgao54@users.noreply.github.com> Date: Tue, 10 Jun 2025 13:02:23 -1000 Subject: [PATCH 11/11] Update docs/integrations/data-ingestion/clickpipes/aws-privatelink.md Co-authored-by: Kunal Gupta <39487888+iamKunalGupta@users.noreply.github.com> --- docs/integrations/data-ingestion/clickpipes/aws-privatelink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md index 50b61cd5b14..65b218e6153 100644 --- a/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md +++ b/docs/integrations/data-ingestion/clickpipes/aws-privatelink.md @@ -66,7 +66,7 @@ Before you can proceed, you'll need to wait for the Resource-Gateway to enter i ```bash aws vpc-lattice get-resource-gateway \ ---resource-gateway-identifier + --resource-gateway-identifier ``` #### 2. Create a VPC Resource-Configuration {#create-resource-configuration}