Implemented TargetAllocator resource deployments. #208

musa-asad · 2024-08-20T05:33:38Z

Description of changes

Updated Dockerfile and Makefile to include the target-allocator.
Implemented prometheus and target-allocator types.
Implemented target-allocator functionality for the amazon-cloudwatch-agent-operator, which does the following:
- If a prometheus configuration is provided alongside enabling targetAllocator in the AmazonCloudWatchAgent custom resource for a given cluster, then the amazon-cloudwatch-agent-operator will build and deploy resources (ConfigMap, Deployment, ServiceAccount, and Service) alongside the amazon-cloudwatch-agent.
Implemented webhook functionality for target-allocator, which assigns a replica for target-allocator to ensure a pod replica is running (currently uses upstream image for target-allocator).
Deploy Prometheus ConfigMap, Volume, and VolumeMount if prometheus configuration is provided; similar to applying https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/prometheus-eks.yaml.
- Implemented service discovery for the prometheus configuration.

Manual testing

Ran make generate && make manifests && make test. [amazon-cloudwatch-agent-operator repo]
Ran make container.
Pushed operator image inside an ECR repo. [AWS console]
Set up an EKS cluster with the CWAgent EKS Add-On.
Ran kubectl edit deployment amazon-cloudwatch-observability-controller-manager -n amazon-cloudwatch and added the operator image. [K8s cluster]
Ran kubectl apply -f config/crd/bases/cloudwatch.aws.amazon.com_amazoncloudwatchagents.yaml --server-side --force-conflicts.
Ran kubectl edit amazoncloudwatchagent cloudwatch-agent -n amazon-cloudwatch and added below:

config: '{"agent":{"region":"us-west-2"}}'
...
mode: statefulset
...
prometheus:
    config:
      scrape_configs:
      - job_name: 'cloudwatch-agent'
        scrape_interval: 10s
        static_configs:
        - targets: [ '0.0.0.0:8888' ]
...
targetAllocator:
  enabled: true
  serviceAccount: cloudwatch-agent

Note: targetAllocatorServiceAcct needs to have permissions set up before we can use it.

ConfigMaps

(when TA is disabled)

(when TA is enabled)

Note: The amazon-cloudwatch-agent package will need to be updated to parse this new prometheus.yaml format.

Deployments

ServiceAccounts

Services

Pods

Prometheus Volume & VolumeMount

Files to be edited/included from upstream in future

Makefile
cmd/otel-allocator

Testing

Implemented unit tests, which pass alongside previous unit tests.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

…stream.

This reverts commit 2c7b788.

This reverts commit ea3c937.

This reverts commit 15a2e2b.

This reverts commit 1235400.

This reverts commit 6da51f6.

internal/manifests/collector/configmap.go

internal/manifests/targetallocator/service.go

okankoAMZ

I focused on changes since my last review. I will take a deeper look after these changes

internal/manifests/collector/config_replace_test.go

internal/manifests/collector/configmap.go

internal/manifests/collector/volume_test.go

internal/manifests/targetallocator/container_test.go

internal/naming/main.go

internal/manifests/collector/collector.go

internal/manifests/targetallocator/service.go

internal/manifests/targetallocator/container.go

internal/manifests/targetallocator/adapters/config_to_prom_config.go

internal/manifests/targetallocator/service.go

okankoAMZ

I think with the provided testing this should work

internal/naming/port.go

mitali-salvi

Could you run the linter on the code ?
make fmt or make goimports

* Adding support for NodeJS auto instrumentation and integ tests (#220) * Support configurable resources for NodeJS. (#225) * Supporting JMX annotations (#240) * Add support for a supplemental YAML configuration for the CloudWatchAgent (#241) * Changed naming for OTLP container ports from agent JSON (#252) * Updated Release Notes for 1.8.0 (#251) * Adjust EKS add-on integration test service count expectations (#256) * Add integration tests for JMX. (#250) * Implemented Target Allocator Container (#214) * Implemented TargetAllocator resource deployments. (#208) * Update cmd/amazon-cloudwatch-agent-target-allocator/config/config.go Co-authored-by: Musa <musaasad@amazon.com> * Update internal/config/main.go Co-authored-by: Musa <musaasad@amazon.com> --------- Co-authored-by: Parampreet Singh <50599809+Paramadon@users.noreply.github.com> Co-authored-by: Musa <musaasad@amazon.com> Co-authored-by: Mitali Salvi <44349099+mitali-salvi@users.noreply.github.com> Co-authored-by: Jeffrey Chien <chienjef@amazon.com>

* Implemented Target Allocator Container (#214) * Merge `main` into `target-allocator` (#232) NodeJS merging-in from main * Supporting K8s 1.31 (#222) (#236) Co-authored-by: Mitali Salvi <44349099+mitali-salvi@users.noreply.github.com> * Implemented TargetAllocator resource deployments. (#208) * Adding target-allocator label to service selectors (#242) * Target allocator TLS Implementation (#239) * Ta https server (#2921) * Added https server, tests, secret marshalling --------- Co-authored-by: ItielOlenick <67790309+ItielOlenick@users.noreply.github.com> * [Target Allocator] Enable Deployment and Daemonset modes for Agent (#253) * Changes error to warning * [CI/CD] Add Target Allocator(TA) Build to Build and Upload Workflow (#247) * edited workflow * Clean up managed resources when disabled (#255) * Reconciler now removes un-used managed resources for CWA collector * remove pprof endpoint (#260) * [TA] One service per Target Allocator (#259) * added one-service per TA * Setup cert-watcher for TA server cert (#264) * [TA] Target Allocator TLS Unit-tests (#265) * TLS tests * Injecting Prometheus path if not specified in agent config (#258) * Injecting Prom path if it doesn't exist * Rebasing Target Allocator Branch to Main (#266) * Adding support for NodeJS auto instrumentation and integ tests (#220) * Support configurable resources for NodeJS. (#225) * Supporting JMX annotations (#240) * Add support for a supplemental YAML configuration for the CloudWatchAgent (#241) * Changed naming for OTLP container ports from agent JSON (#252) * Updated Release Notes for 1.8.0 (#251) * Adjust EKS add-on integration test service count expectations (#256) * Add integration tests for JMX. (#250) * Implemented Target Allocator Container (#214) * Implemented TargetAllocator resource deployments. (#208) * Update cmd/amazon-cloudwatch-agent-target-allocator/config/config.go Co-authored-by: Musa <musaasad@amazon.com> * Update internal/config/main.go Co-authored-by: Musa <musaasad@amazon.com> --------- Co-authored-by: Parampreet Singh <50599809+Paramadon@users.noreply.github.com> Co-authored-by: Musa <musaasad@amazon.com> Co-authored-by: Mitali Salvi <44349099+mitali-salvi@users.noreply.github.com> Co-authored-by: Jeffrey Chien <chienjef@amazon.com> --------- Co-authored-by: Musa <musaasad@amazon.com> Co-authored-by: Mitali Salvi <44349099+mitali-salvi@users.noreply.github.com> Co-authored-by: ItielOlenick <67790309+ItielOlenick@users.noreply.github.com> Co-authored-by: Kaushik Surya <108111936+sky333999@users.noreply.github.com> Co-authored-by: Parampreet Singh <50599809+Paramadon@users.noreply.github.com> Co-authored-by: Jeffrey Chien <chienjef@amazon.com>

musa-asad added 2 commits August 19, 2024 23:19

Implemented TargetAllocator deployment and other dependencies from up…

4b1eb31

…stream.

Added unit tests, organization, and relevant build files.

0e079bd

musa-asad self-assigned this Aug 20, 2024

musa-asad added 25 commits August 20, 2024 01:53

Fixed tests.

ca26265

Update crds.

3a871af

Updated docs.

0755622

Reversed crd changes.

a77533f

Generated files.

1d8a923

Updated webhooks.

15a2e2b

Updated config_replace.go to include allocator logic.

2c7b788

Revert "Updated config_replace.go to include allocator logic."

4202fae

This reverts commit 2c7b788.

Updated doc and imports.

ea3c937

Revert "Updated doc and imports."

0ef5529

This reverts commit ea3c937.

Revert "Updated webhooks."

ec14fe5

This reverts commit 15a2e2b.

Replica if enabled.

d324ccf

Auto-generated.

2979609

Added default config.

1235400

Revert "Added default config."

db8c52f

This reverts commit 1235400.

Removed errors for testing.

6da51f6

Revert "Removed errors for testing."

36d1c6c

This reverts commit 6da51f6.

Use prometheusConfig.

3cdebd9

Fix version.

cfe2dbe

Included validation in webhooks.

c220dd5

Use prometheus config.

561b2ff

Updated webook test and added types to v1alpha2.

60a2856

Fixed test.

f8720f4

Removed extra comma.

75a0168

Auto-generated files.

a1860bd

musa-asad requested review from mitali-salvi and okankoAMZ August 21, 2024 18:42

musa-asad commented Sep 27, 2024

View reviewed changes

internal/manifests/collector/configmap.go Outdated Show resolved Hide resolved

Remove comment.

557c263

mitali-salvi reviewed Sep 27, 2024

View reviewed changes

internal/manifests/collector/configmap.go Outdated Show resolved Hide resolved

internal/manifests/targetallocator/service.go Outdated Show resolved Hide resolved

Use ConfigMapEntry().

9ec6ae7

okankoAMZ reviewed Sep 27, 2024

View reviewed changes

Use function for service name.

f3446ec

musa-asad requested review from okankoAMZ and mitali-salvi September 27, 2024 17:18

musa-asad added 4 commits September 27, 2024 13:34

Use constants isntead of ConfigMapEntry().

acd158f

Use function for TAService().

4946b79

Use ConfigMapEntry() with fixed tests.

801e9eb

Test ConfigMapEntry() for TA.

2519ad1

okankoAMZ requested changes Sep 27, 2024

View reviewed changes

internal/manifests/targetallocator/service.go Outdated Show resolved Hide resolved

internal/manifests/targetallocator/container.go Outdated Show resolved Hide resolved

okankoAMZ reviewed Sep 27, 2024

View reviewed changes

internal/manifests/targetallocator/adapters/config_to_prom_config.go Outdated Show resolved Hide resolved

internal/manifests/targetallocator/service.go Outdated Show resolved Hide resolved

Made target port 8080.

deb08d5

musa-asad requested a review from okankoAMZ September 27, 2024 20:15

okankoAMZ previously approved these changes Sep 27, 2024

View reviewed changes

internal/naming/port.go Outdated Show resolved Hide resolved

internal/naming/port.go Outdated Show resolved Hide resolved

Addressed nits.

ae90c31

musa-asad dismissed okankoAMZ’s stale review via ae90c31 September 27, 2024 22:14

musa-asad requested a review from okankoAMZ September 27, 2024 22:14

okankoAMZ previously approved these changes Sep 30, 2024

View reviewed changes

mitali-salvi reviewed Oct 1, 2024

View reviewed changes

Fix imports.

1986ad7

musa-asad dismissed okankoAMZ’s stale review via 1986ad7 October 1, 2024 19:51

okankoAMZ approved these changes Oct 1, 2024

View reviewed changes

mitali-salvi approved these changes Oct 1, 2024

View reviewed changes

musa-asad merged commit b3cf3cf into aws:target-allocator Oct 1, 2024

okankoAMZ pushed a commit to okankoAMZ/amazon-cloudwatch-agent-operator that referenced this pull request Oct 31, 2024

Implemented TargetAllocator resource deployments. (aws#208)

332dc51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Implemented TargetAllocator resource deployments. #208

Implemented TargetAllocator resource deployments. #208

Uh oh!

musa-asad commented Aug 20, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

okankoAMZ left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

okankoAMZ left a comment

Uh oh!

Uh oh!

Uh oh!

mitali-salvi left a comment

Uh oh!

Uh oh!

Implemented TargetAllocator resource deployments. #208

Implemented TargetAllocator resource deployments. #208

Uh oh!

Conversation

musa-asad commented Aug 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Manual testing

Files to be edited/included from upstream in future

Testing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

okankoAMZ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

okankoAMZ left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mitali-salvi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

musa-asad commented Aug 20, 2024 •

edited

Loading