-
Notifications
You must be signed in to change notification settings - Fork 338
Add docs for AWS PrivateLink using VPC Resource #3925
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
d68b642
9e17965
6ef914c
7872adc
2d1be58
a7546b0
9ee8161
d343ca5
7351b42
ef8cd8b
ac19421
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -32,26 +32,96 @@ | |
- [MSK multi-VPC connectivity for MSK ClickPipe](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) | ||
- [VPC endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html) | ||
|
||
Follow the links above for detailed instructions on how to set up the respective AWS PrivateLink shares. | ||
|
||
### VPC resource {#vpc-resource} | ||
|
||
Your VPC resources can be accessed in ClickPipes using PrivateLink. | ||
Your VPC resources can be accessed in ClickPipes using PrivateLink and [AWS VPC Lattice](https://docs.aws.amazon.com/vpc-lattice/latest/ug/what-is-vpc-lattice.html). This approach doesn't require setting up a load balancer in front of your data source. | ||
|
||
Resource configuration can be targeted with a specific host or RDS cluster ARN. | ||
Cross-region is not supported. | ||
|
||
It's the preferred choice for Postgres CDC ingesting data from an RDS cluster. | ||
|
||
See a [getting started](https://docs.aws.amazon.com/vpc/latest/privatelink/resource-configuration.html) guide for more details. | ||
To set up PrivateLink with VPC resource: | ||
1. Create a resource gateway | ||
2. Create a resource configuration | ||
3. Create a resource share | ||
|
||
:::info | ||
VPC resource needs to be shared with a ClickPipes account. Add `072088201116` to the allowed principals to your resource share configuration. | ||
See AWS guide for [sharing resources](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) for more details. | ||
::: | ||
#### 1. Create a Resource-Gateway {#create-resource-gateway} | ||
|
||
Resource-Gateway is the point that receives traffic for specified resources in your VPC. | ||
|
||
You can create a Resource-Gateway from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-gateway.html) or with the following command: | ||
|
||
```bash | ||
aws vpc-lattice create-resource-gateway \ | ||
--vpc-identifier <VPC_ID> \ | ||
--subnet-ids <SUBNET_IDS> \ | ||
--security-group-ids <SG_IDs> \ | ||
--name <RESOURCE_GATEWAY_NAME> | ||
``` | ||
|
||
The output will contain a Resource-Gateway id, which you will need for the next step. | ||
|
||
Before you can proceed, you'll need to wait for the Resource-Gateway to enter into an `Active` state. You can check the state by running the following command: | ||
|
||
```bash | ||
aws vpc-lattice get-resource-gateway \ | ||
--resource-gateway-identifier <RESOURCE_GATEWAY_ID> | ||
``` | ||
|
||
#### 2. Create a VPC Resource-Configuration {#create-resource-configuration} | ||
|
||
Resource-Configuration is associated with Resource-Gateway to make your resource accessible. | ||
|
||
You can create a Resource-Configuration from the [AWS console](https://docs.aws.amazon.com/vpc/latest/privatelink/create-resource-configuration.html) or with the following command: | ||
|
||
```bash | ||
aws vpc-lattice create-resource-configuration \ | ||
--resource-gateway-identifier <RESOURCE_GATEWAY_ID> \ | ||
--type <RESOURCE_CONFIGURATION_TYPE> \ | ||
--resource-configuration-definition <RESOURCE_CONFIGURATION_DEFINITION> \ | ||
--name <RESOURCE_CONFIGURATION_NAME> | ||
``` | ||
|
||
The simplest [resource configuration type](https://docs.aws.amazon.com/vpc-lattice/latest/ug/resource-configuration.html#resource-configuration-types) is a single Resource-Configuration. You can configure with the ARN directly, or share an IP address or a domain name that is publicly resolvable. | ||
|
||
For example, to configure with the ARN of an RDS Cluster: | ||
|
||
```bash | ||
aws vpc-lattice create-resource-configuration \ | ||
--name my-rds-cluster-config \ | ||
--type ARN \ | ||
--resource-gateway-identifier rgw-0bba03f3d56060135 \ | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. arbitrary value taken from aws doc |
||
--resource-configuration-definition 'arnResource={arn=arn:aws:rds:us-east-1:123456789012:cluster:my-rds-cluster}' | ||
``` | ||
|
||
jgao54 marked this conversation as resolved.
Show resolved
Hide resolved
|
||
The output will contain a Resource-Configuration ARN, which you will need for the next step. It will also contain a Resource-Configuration ID, which you will need to set up a ClickPipe connection with VPC resource. | ||
|
||
#### 3. Create a Resource-Share {#create-resource-share} | ||
|
||
Sharing your resource requires a Resource-Share. This is facilitated through the Resource Access Manager (RAM). | ||
|
||
You can put the Resource-Configuration into the Resource-Share through [AWS console](https://docs.aws.amazon.com/ram/latest/userguide/working-with-sharing-create.html) or by running the following command with ClickPipes account ID `072088201116`: | ||
|
||
```bash | ||
aws ram create-resource-share \ | ||
--principals 072088201116 \ | ||
--resource-arns <RESOURCE_CONFIGURATION_ARN> \ | ||
--name <RESOURCE_SHARE_NAME> | ||
``` | ||
|
||
The output will contain a Resource-Share ARN, which you will need to set up a ClickPipe connection with VPC resource. | ||
|
||
You are ready to [create a ClickPipe with Reverse private endpoint](#creating-clickpipe) using VPC resource. You will need to: | ||
- Set `VPC endpoint type` to `VPC Resource`. | ||
- Set `Resource configuration ID` to the ID of the Resource-Configuration created in step 2. | ||
- Set `Resource share ARN` to the ARN of the Resource-Share created in step 3. | ||
|
||
For more details on PrivateLink with VPC resource, see [AWS documentation](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-access-resources.html). | ||
|
||
### MSK multi-VPC connectivity {#msk-multi-vpc} | ||
|
||
The MSK multi-VPC is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster. | ||
The [Multi-VPC connectivity](https://docs.aws.amazon.com/msk/latest/developerguide/aws-access-mult-vpc.html) is a built-in feature of AWS MSK that allows you to connect multiple VPCs to a single MSK cluster. | ||
Check notice on line 124 in docs/integrations/data-ingestion/clickpipes/aws-privatelink.md
|
||
Private DNS support is out of the box and does not require any additional configuration. | ||
Cross-region is not supported. | ||
|
||
|
@@ -67,7 +137,7 @@ | |
|
||
### VPC endpoint service {#vpc-endpoint-service} | ||
|
||
VPC service is another approach to share your data source with ClickPipes. | ||
[VPC endpoint service](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html) is another approach to share your data source with ClickPipes. | ||
It requires setting up a NLB (Network Load Balancer) in front of your data source | ||
and configuring the VPC endpoint service to use the NLB. | ||
|
||
|
Uh oh!
There was an error while loading. Please reload this page.