chore: org+k8s+cloudtrail-s3-sqs+manual input roles (#32)

Author: iru

.github/workflows/ci-integration-tests.yaml

@@ -16,7 +16,7 @@ concurrency: terraform
 
 jobs:
   integration_test_ecs:
-    concurrency: terraform-account
+#    concurrency: terraform-account
 
     name: Test-Kitchen-ECS
     runs-on: ubuntu-latest
@@ -66,7 +66,7 @@ jobs:
         run: bundle exec kitchen destroy "organizational-aws"
 
   integration_test-eks:
-    concurrency: terraform-account
+#    concurrency: terraform-account
     continue-on-error: true
 
     name: Test-Kitchen-EKS
@@ -109,7 +109,7 @@ jobs:
       - name: Destroy single-account-k8s resources
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCESS_KEY_ID }}
-          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}}}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}
           AWS_REGION: ${{ secrets.AWS_REGION }}
         if: ${{ failure() }}
         run: bundle exec kitchen destroy "single-account-k8s-aws"

.github/workflows/ci-test-cleanup.yaml

@@ -22,9 +22,12 @@ jobs:
         with:
           ruby-version: 2.7
           bundler-cache: true
-
+      - name: Create kind cluster # this is not really needed but kitchen requires it
+        uses: helm/kind-action@v1.2.0
+        with:
+          wait: 120s
       - name: Destroy resources
-        run: bundle exec kitchen destroy
+        run: bundle exec kitchen destroy single
 
   cleanup-org:
     name: Test Cleanup Org
@@ -44,6 +47,9 @@ jobs:
         with:
           ruby-version: 2.7
           bundler-cache: true
-
+      - name: Create kind cluster # this is not really needed but kitchen requires it
+        uses: helm/kind-action@v1.2.0
+        with:
+          wait: 120s
       - name: Destroy resources
-        run: bundle exec kitchen destroy
+        run: bundle exec kitchen destroy organizational

.kitchen.yml

@@ -20,3 +20,6 @@ suites:
   - name: organizational
     driver:
       root_module_directory: test/fixtures/organizational
+  - name: organizational-k8s
+    driver:
+      root_module_directory: test/fixtures/organizational-k8s

CONTRIBUTE.md

@@ -64,8 +64,8 @@ $ bundle exec kitchen tests
 Because CI/CD sometimes fail, we setup the Terraform state to be handled in backend (s3+dynamo) within the Sysdig AWS backend (sysdig-test-account).
 In order to be able to use this Terraform backend AWS credentials are configured as Github project secret
 
-If you need to handle the remote state on your local for any cleanup, please do it using `kitchen destroy`, not `terraform destroy`
-
+If terraform state ends up in bad shape and not cleaned, use the action called `Test Cleanup` that should destroy any messed situation.
+If this does not work, try it from your local, but please do it using `kitchen destroy`, not `terraform destroy` unless you really know what you're doing :]
 
 ### Deployed infrastructure resources
 

examples-internal/organizational-k8s-threat-reuse_cloudtrail/README.md

@@ -0,0 +1,133 @@
+# Sysdig Secure for Cloud in AWS<br/>:: Organizational, threat-detection with pre-existing resources (EKS + cloudtrail through S3-SNS-SQS events)
+
+
+- Sysdig **Helm** chart will be used to deploy threat-detection
+    - [Cloud-Connector Chart](https://charts.sysdig.com/charts/cloud-connector/)
+    - This charts requires specific AWS credentials to be passed by parameter (accessKeyId and secretAccessKey)
+- An existing cloudtrail is used, but instead of sending events directly to an SNS topic (disabled), we will make use of a topic (SQS)
+  which will be subscribed to the multiple possible SNS topics listening to the cloudtrail-S3 bucket changes.
+
+![diagram](https://github.com/sysdiglabs/terraform-aws-secure-for-cloud/blob/master/examples-internal/organizational-k8s-threat-reuse_cloudtrail/diagram.png)
+
+All the required resources and workloads will be run under the same AWS account, held in a member-account of the organization.
+
+## Prerequisites
+
+Minimum requirements:
+
+1. **AWS** profile credentials configured within yor `aws` provider
+2. A **Kubernetes** cluster configured within your `helm` provider
+3. **Sysdig** Secure API token , as input variable value
+    ```
+    sysdig_secure_api_token=<SECURE_API_TOKEN>
+    ```
+4. S3 event-notification subscribed SNS topic(s).<br/>see `modules/infrastructure/cloudtrail_s3-sns-sqs` for guidance<br/><br/>
+5. **SQS topic** subscribed to the S3-SNS event notifications.<br/>The ARN of this SQS will be used as an input parameter to the module.<br/>
+   see `modules/infrastructure/sqs-sns-subscription` for guidance`<br/><br/>
+6. If the module is to be deployed on an AWS Organization **member account** which is not the same where the Cloudtrail-S3 events are located,
+   the `organization_managed_role_arn` input variable must be used<br/>
+   This will provide the **ARN of a role** that `cloud-connector` module will use to fetch the events from the S3 bucket.<br/>
+   see `modules/infrastructure/permissions/eks-org-role` for guidance`<br/><br/>
+
+## Usage
+
+For quick testing, use this snippet on your terraform files.
+
+```terraform
+provider "aws" {
+  region = var.region
+  ...
+}
+
+provider "helm" {
+  ...
+}
+
+module "org_k8s_threat_reuse_cloudtrail" {
+  source = "sysdiglabs/secure-for-cloud/aws//examples-internal/organizational-k8s-threat-reuse_cloudtrail"
+
+  sysdig_secure_api_token   = "00000000-1111-2222-3333-444444444444"
+
+  region                          = "CLOUDTRAIL_SNS_SQS_REGION"
+  cloudtrail_s3_sns_sqs_url       = "SQS-URL"
+  organization_managed_role_arn   = "ARN_ROLE_FOR_MEMBER_ACCOUNT_PERMISSIONS"
+
+  aws_access_key_id         = "AWS_ACCESSK_KEY"
+  aws_secret_access_key     = "AWS_SECRET_ACCESS_KEY"
+}
+
+```
+
+See [inputs summary](#inputs) or module module [`variables.tf`](https://github.com/sysdiglabs/terraform-aws-secure-for-cloud/blob/master/examples-internal/organizational-k8s-threat-reuse_cloudtrail/variables.tf) file for more optional configuration.
+
+To run this example you need have your [aws account profile configured in CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) and to execute:
+```terraform
+$ terraform init
+$ terraform plan
+$ terraform apply
+```
+
+Notice that:
+* This example will create resources that cost money.<br/>Run `terraform destroy` when you don't need them anymore
+* All created resources will be created within the tags `product:sysdig-secure-for-cloud`, within the resource-group `sysdig-secure-for-cloud`
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Requirements
+
+| Name | Version |
+|------|---------|
+| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.15.0 |
+| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 3.50.0 |
+| <a name="requirement_helm"></a> [helm](#requirement\_helm) | >=2.3.0 |
+
+## Providers
+
+| Name | Version |
+|------|---------|
+| <a name="provider_helm"></a> [helm](#provider\_helm) | >=2.3.0 |
+
+## Modules
+
+| Name | Source | Version |
+|------|--------|---------|
+| <a name="module_resource_group"></a> [resource\_group](#module\_resource\_group) | ../../modules/infrastructure/resource-group |  |
+| <a name="module_ssm"></a> [ssm](#module\_ssm) | ../../modules/infrastructure/ssm |  |
+
+## Resources
+
+| Name | Type |
+|------|------|
+| [helm_release.cloud_connector](https://registry.terraform.io/providers/hashicorp/helm/latest/docs/resources/release) | resource |
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| <a name="input_aws_access_key_id"></a> [aws\_access\_key\_id](#input\_aws\_access\_key\_id) | cloud-connector. aws credentials in order to access required aws resources. aws.accessKeyId | `string` | n/a | yes |
+| <a name="input_aws_secret_access_key"></a> [aws\_secret\_access\_key](#input\_aws\_secret\_access\_key) | cloud-connector. aws credentials in order to access required aws resources. aws.secretAccessKey | `string` | n/a | yes |
+| <a name="input_cloudtrail_s3_sns_sqs_url"></a> [cloudtrail\_s3\_sns\_sqs\_url](#input\_cloudtrail\_s3\_sns\_sqs\_url) | Organization cloudtrail event notification  S3-SNS-SQS URL to listen to | `string` | n/a | yes |
+| <a name="input_sysdig_secure_api_token"></a> [sysdig\_secure\_api\_token](#input\_sysdig\_secure\_api\_token) | Sysdig Secure API token | `string` | n/a | yes |
+| <a name="input_name"></a> [name](#input\_name) | Name to be assigned to all child resources. A suffix may be added internally when required. Use default value unless you need to install multiple instances | `string` | `"sfc"` | no |
+| <a name="input_organization_managed_role_arn"></a> [organization\_managed\_role\_arn](#input\_organization\_managed\_role\_arn) | `sysdig_secure_for_cloud_role_arn` for cloud-connector assumeRole in order to read cloudtrail s3 events</li><li>and the `connector_ecs_task_role_name` which has been granted trusted-relationship over the secure\_for\_cloud\_role | `string` | `"none"` | no |
+| <a name="input_region"></a> [region](#input\_region) | Default region for resource creation in both organization master and secure-for-cloud member account | `string` | `"eu-central-1"` | no |
+| <a name="input_sysdig_secure_endpoint"></a> [sysdig\_secure\_endpoint](#input\_sysdig\_secure\_endpoint) | Sysdig Secure API endpoint | `string` | `"https://secure.sysdig.com"` | no |
+| <a name="input_tags"></a> [tags](#input\_tags) | sysdig secure-for-cloud tags | `map(string)` | <pre>{<br>  "product": "sysdig-secure-for-cloud"<br>}</pre> | no |
+
+## Outputs
+
+No outputs.
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+
+
+## Troubleshooting
+
+- Q1: When I deploy it, cloud-connector gives an error saying `api error AWS.SimpleQueueService.NonExistentQueue: The specified queue does not exist for this wsdl version`
+  S1: make use of the `var.region` to specify where the resources are on the organzation managed account (sqs)
+
+## Authors
+
+Module is maintained and supported by [Sysdig](https://sysdig.com).
+
+## License
+
+Apache 2 Licensed. See LICENSE for full details.

examples-internal/organizational-k8s-threat-reuse_cloudtrail/cloud-connector.tf

@@ -0,0 +1,52 @@
+resource "helm_release" "cloud_connector" {
+
+  provider = helm
+
+  name = "cloud-connector"
+
+  repository = "https://charts.sysdig.com"
+  chart      = "cloud-connector"
+
+  create_namespace = true
+  namespace        = var.name
+
+  set {
+    name  = "image.pullPolicy"
+    value = "Always"
+  }
+
+  set_sensitive {
+    name  = "sysdig.secureAPIToken"
+    value = var.sysdig_secure_api_token
+  }
+
+  set_sensitive {
+    name  = "aws.accessKeyId"
+    value = var.aws_access_key_id
+  }
+
+  set_sensitive {
+    name  = "aws.secretAccessKey"
+    value = var.aws_secret_access_key
+  }
+
+  set {
+    name  = "aws.region"
+    value = var.region
+  }
+
+  set {
+    name  = "sysdig.url"
+    value = var.sysdig_secure_endpoint
+  }
+
+  values = [
+    <<CONFIG
+logging: info
+ingestors:
+  - aws-cloudtrail-s3-sns-sqs:
+      queueURL: ${var.cloudtrail_s3_sns_sqs_url}
+      assumeRole: ${var.organization_managed_role_arn}
+CONFIG
+  ]
+}

examples-internal/organizational-k8s-threat-reuse_cloudtrail/diagram.png

@@ -0,0 +1,66 @@
+# diagrams as code vía https://diagrams.mingrammer.com
+from diagrams import Cluster, Diagram, Edge, Node
+from diagrams.aws.compute import EKS
+from diagrams.aws.general import General
+from diagrams.aws.integration import SNS, SQS
+from diagrams.aws.management import Cloudtrail
+from diagrams.aws.security import IAM, IAMRole
+from diagrams.aws.storage import S3
+from diagrams.custom import Custom
+
+from diagrams.k8s.group import Namespace
+from diagrams.k8s.compute import Deployment
+
+diagram_attr = {
+    "pad":"0.25"
+}
+
+role_attr = {
+   "imagescale":"false",
+   "height":"1.5",
+   "width":"3",
+   "fontsize":"9",
+}
+
+color_event="firebrick"
+color_scanning = "dark-green"
+color_permission="red"
+color_creates="darkblue"
+color_non_important="gray"
+color_sysdig="lightblue"
+
+
+
+with Diagram("Sysdig Secure for Cloud{}(org-threat_detection-k8s-cloudtrail_s3_sns_sqs-eks)".format("\n"), graph_attr=diagram_attr, filename="diagram", show=True, direction="TB"):
+
+    with Cluster("AWS account (sysdig)"):
+        sds = Custom("Sysdig Secure", "../../resources/diag-sysdig-icon.png")
+
+    with Cluster("AWS organization"):
+
+        with Cluster("member accounts (main target)", graph_attr={"bgcolor":"lightblue"}):
+#             resources = General("resources-1..n\n(events)")
+
+            with Cluster("sysdig-secure-for-cloud resources"):
+                eks = EKS("EKS\n(pre-existing)")
+                with Cluster("namespace: sfc"):
+                    cc_deployment = Deployment("cloud-connector")
+                    eks_deployments = [cc_deployment]
+
+
+        with Cluster("management account"):
+#             resources2 = General("resources-1..n\n(events)")
+
+            cloudtrail          = Cloudtrail("cloudtrail\n(organizational)", shape="plaintext")
+            cloudtrail_s3       = S3("cloudtrail-s3-events")
+            sns                 = [SNS("sns /path-1"), SNS("sns /path-2"), SNS("sns /path-n")]
+            sqs                 = SQS("sqs")
+            cloudtrail >> Edge(color=color_event) >> cloudtrail_s3 >> Edge(color=color_event) >> sns << sqs
+#             resources2 >> Edge(color=color_event, style="dashed") >>  cloudtrail
+
+            management_credentials  = IAM("credentials", fontsize="10")
+
+        cc_deployment >>  Edge(color=color_event, style="dashed", label="subscribed") >> sqs
+#         resources >> Edge(color=color_event, style="dashed") >>  cloudtrail
+
+    cc_deployment >> Edge(color=color_sysdig) >> sds

examples-internal/organizational-k8s-threat-reuse_cloudtrail/diagram.py

@@ -0,0 +1,15 @@
+#-------------------------------------
+# general resources
+#-------------------------------------
+
+module "resource_group" {
+  source = "../../modules/infrastructure/resource-group"
+  name   = var.name
+  tags   = var.tags
+}
+
+module "ssm" {
+  source                  = "../../modules/infrastructure/ssm"
+  name                    = var.name
+  sysdig_secure_api_token = var.sysdig_secure_api_token
+}