chore(refact): module and permission optionals (#35)

* refact. module optionals for examples/single-account-k8s
* refact. permissions/single-account-user into several modules > iam-credentials, general cloud-connector, 
* refact. /permissions/org-management-role to /permissions/ecs-org-rolecloud-scanning

* test: added test with minikube kind

Author: iru

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,7 +1,7 @@
 > PR template
 >
 > * for a cleaner PR, **delete whatever is not required**
-> * use pull-request **drafts for visiblity on WIP branches**
+> * use pull-request **drafts for visibility on WIP branches**
 > * unless a revision is desired in order to validate, or gather some feedback, **you are free to merge it as long as**
 >   * validation checkers are all green-lighted
 >   * pre-merge checklist has been reviewed. for more detail check **`/CONTRIBUTE.md`**

.github/workflows/ci-integration-tests.yaml

@@ -15,8 +15,10 @@ on:
 concurrency: terraform
 
 jobs:
-  integration_test:
-    name: Test-Kitchen
+  integration_test_ecs:
+    concurrency: terraform-account
+
+    name: Test-Kitchen-ECS
     runs-on: ubuntu-latest
     env:
       TF_VAR_sysdig_secure_endpoint: https://secure.sysdig.com
@@ -36,25 +38,23 @@ jobs:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}
           AWS_REGION: ${{ secrets.AWS_REGION }}
-        run: bundle exec kitchen test single-account
+        run: bundle exec kitchen test "single-account-aws"
 
       - name: Destroy single-account resources
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}
           AWS_REGION: ${{ secrets.AWS_REGION }}
         if: ${{ failure() }}
-        run: bundle exec kitchen destroy single-account
-
-
+        run: bundle exec kitchen destroy "single-account-aws"
 
       - name: Run organizational test
         env:
           AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_MANAGED_ACCESS_KEY_ID }}
           AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_MANAGED_SECRET_ACCESS_KEY }}
           AWS_REGION: ${{ secrets.AWS_REGION }}
           TF_VAR_sysdig_secure_for_cloud_member_account_id: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCOUNT_ID }}
-        run: bundle exec kitchen test organizational
+        run: bundle exec kitchen test "organizational-aws"
 
       - name: Destroy organizational resources
         env:
@@ -63,4 +63,53 @@ jobs:
           AWS_REGION: ${{ secrets.AWS_REGION }}
           TF_VAR_sysdig_secure_for_cloud_member_account_id: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCOUNT_ID }}
         if: ${{ failure() }}
-        run: bundle exec kitchen destroy organizational
+        run: bundle exec kitchen destroy "organizational-aws"
+
+  integration_test-eks:
+    concurrency: terraform-account
+    continue-on-error: true
+
+    name: Test-Kitchen-EKS
+    runs-on: ubuntu-latest
+    env:
+      TF_VAR_sysdig_secure_endpoint: https://secure.sysdig.com
+      TF_VAR_sysdig_secure_api_token: ${{secrets.KUBELAB_SECURE_API_TOKEN}}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: 2.7
+          bundler-cache: true
+
+      - name: Create kind cluster
+        uses: helm/kind-action@v1.2.0
+        with:
+          wait: 120s
+
+      - name: Run single-account-k8s test
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}
+          AWS_REGION: ${{ secrets.AWS_REGION }}
+        run: bundle exec kitchen test "single-account-k8s-aws"
+
+      - name: Inspect k8s failures
+        if: ${{ failure() }}
+        run: |
+          kubectl get namespaces
+          kubectl get deployments -n sfc-tests-kitchen-singlek8s
+          kubectl describe deployment cloud-connector -n sfc-tests-kitchen-singlek8s
+          kubectl logs deployment.apps/cloud-connector -n sfc-tests-kitchen-singlek8s
+          kubectl logs deployment.apps/cloud-scanning -n sfc-tests-kitchen-singlek8s
+
+
+      - name: Destroy single-account-k8s resources
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}}}
+          AWS_REGION: ${{ secrets.AWS_REGION }}
+        if: ${{ failure() }}
+        run: bundle exec kitchen destroy "single-account-k8s-aws"

.github/workflows/ci-test-cleanup.yaml

@@ -2,17 +2,40 @@ name: CI - Test Cleanup
 on:
   workflow_dispatch
 
+concurrency: terraform
+
 jobs:
-  test_cleanup:
+  cleanup-cloudnative:
     name: Test Cleanup
     runs-on: ubuntu-latest
     env:
-      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_ACCESS_KEY_ID }}
-      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_SECRET_ACCESS_KEY }}
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_CLOUDNATIVE_SECRET_ACCESS_KEY }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       TF_VAR_sysdig_secure_endpoint: https://secure.sysdig.com
       TF_VAR_sysdig_secure_api_token: ${{secrets.KUBELAB_SECURE_API_TOKEN}}
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+
+      - uses: ruby/setup-ruby@v1
+        with:
+          ruby-version: 2.7
+          bundler-cache: true
 
+      - name: Destroy resources
+        run: bundle exec kitchen destroy
+
+  cleanup-org:
+    name: Test Cleanup Org
+    runs-on: ubuntu-latest
+    env:
+      AWS_ACCESS_KEY_ID: ${{ secrets.AWS_QA_MANAGED_ACCESS_KEY_ID }}
+      AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_QA_MANAGED_SECRET_ACCESS_KEY }}
+      AWS_REGION: ${{ secrets.AWS_REGION }}
+      TF_VAR_sysdig_secure_for_cloud_member_account_id: ${{ secrets.AWS_QA_CLOUDNATIVE_ACCOUNT_ID }}
+      TF_VAR_sysdig_secure_endpoint: https://secure.sysdig.com
+      TF_VAR_sysdig_secure_api_token: ${{secrets.KUBELAB_SECURE_API_TOKEN}}
     steps:
       - name: Checkout
         uses: actions/checkout@v2
@@ -23,5 +46,4 @@ jobs:
           bundler-cache: true
 
       - name: Destroy resources
-        if: ${{ failure() }}
         run: bundle exec kitchen destroy

.kitchen.yml

@@ -14,6 +14,9 @@ suites:
   - name: single-account
     driver:
       root_module_directory: test/fixtures/single-account
+  - name: single-account-k8s
+    driver:
+      root_module_directory: test/fixtures/single-account-k8s
   - name: organizational
     driver:
       root_module_directory: test/fixtures/organizational

.pre-commit-config.yaml

@@ -27,7 +27,7 @@ repos:
         args:
           - '--args=--sort-by required'
       - id: terraform_tflint
-        exclude: test\/.*$
+        exclude: (test)|(examples-internal)\/.*$
         args:
           - '--args=--only=terraform_deprecated_interpolation'
           - '--args=--only=terraform_deprecated_index'

CONTRIBUTE.md

@@ -16,14 +16,16 @@
 
 # Pull Request
 
+Should any pre-merge test fail, check `/.github/workflows/ci-integration-test.yaml` to identify what's required
+
 ## 1. Check::Pre-Commit
 
 Technical validation for terraform **lint**, **validation**, and **documentation**
 
 We're using **pre-commit** |  https://pre-commit.com
   - Defined in `/.pre-commit-config.yaml`
   - custom configuration | https://github.com/sysdiglabs/terraform-google-secure-for-cloud/blob/master/.pre-commit-config.yaml
-  - current `terraform-docs` requires developer to create `README.md` file, with the enclosure tags for docs to insert the automated content
+  - current `terraform-docs` version, requires developer to create `README.md` file, with the enclosure tags for docs to insert the automated content
   ```markdown
   <!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
   <!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
@@ -35,8 +37,6 @@ Final user validation. Checks that the snippets for the usage, stated in the off
 
 Implemented vía **Terraform Kitchen** | https://newcontext-oss.github.io/kitchen-terraform
 
-- Defined in `/.github/workflows/ci-integration-test.yaml`.
-
 ### Kitchen
 
 - Kitchen configuration can be found in `/.kitchen.yml`
@@ -59,20 +59,18 @@ $ bundle exec kitchen tests
 
 ```
 
-
-
-
 ### Terraform Backend
 
 Because CI/CD sometimes fail, we setup the Terraform state to be handled in backend (s3+dynamo) within the Sysdig AWS backend (sysdig-test-account).
 In order to be able to use this Terraform backend AWS credentials are configured as Github project secret
 
+If you need to handle the remote state on your local for any cleanup, please do it using `kitchen destroy`, not `terraform destroy`
+
 
 ### Deployed infrastructure resources
 
 Check project github secrets for clarification
 
-
 # Release
 
 - Use **semver** for releases https://semver.org

Gemfile

@@ -1,6 +1,6 @@
 source "https://rubygems.org/"
 
 gem "kitchen-terraform", "~> 6.0.0"
-gem 'aws-sdk', '~> 3.0.1'
-gem 'awspec', '~> 1.24.0'
-gem 'kitchen-verifier-awspec', '~> 0.2.0'
+#gem 'aws-sdk', '~> 3.0.1'
+#gem 'awspec', '~> 1.24.0'
+#gem 'kitchen-verifier-awspec', '~> 0.2.0'