Unable to send data to aws cloudwatch log.

[Strange21]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

I am trying to send the data to AWS cloudwatch log. So firstly I have tried sending data from stdin (configured my source as stdin), and I am able to send the data after executing command
echo "test message" | vector
Following is the logs for above command

2023-11-01T08:23:16.344503Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-11-01T08:23:16.345115Z  WARN vector::config::loading: DEPRECATED Using the deprecated "/etc/vector/vector.toml" config path as the default config location. Vector is migrating to YAML as the default config format. Future Vector versions will use "/etc/vector/vector.yaml" as the default config location.
2023-11-01T08:23:16.345169Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-11-01T08:23:16.346780Z  INFO vector::sources::file_descriptors: Capturing stdin.
2023-11-01T08:23:16.417517Z  INFO vector::topology::running: Running healthchecks.
2023-11-01T08:23:16.417612Z  INFO vector: Vector has started. debug="false" version="0.33.0" arch="x86_64" revision="89605fb 2023-09-27 14:18:24.180809939"
2023-11-01T08:23:16.417627Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-11-01T08:23:16.417964Z  INFO vector_common::shutdown: All sources have finished.
2023-11-01T08:23:16.417981Z  INFO vector_common::shutdown: All sources have finished.
2023-11-01T08:23:16.419058Z  INFO vector::app: All sources have finished.
2023-11-01T08:23:16.419077Z  INFO vector: Vector has stopped.
2023-11-01T08:23:16.419284Z  INFO sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=1
2023-11-01T08:23:16.420619Z  INFO vector::topology::running: Shutting down... Waiting on running components. remaining_components="aws_cloudwatch_logs" time_remaining="59 seconds left"
2023-11-01T08:23:17.010537Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: service error component_kind="sink" component_type="aws_cloudwatch_logs" component_id=aws_cloudwatch_logs component_name=aws_cloudwatch_logs
2023-11-01T08:23:17.058468Z  INFO sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Putting logs. token=Some("49039859565611948875833551386623940985914745056819434304")
2023-11-01T08:23:17.493723Z  INFO sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::request: Putting logs was successful. next_token=Some("49645815922851066143980076859732739576820567984046080434")

No when I try to send the data from http_server with following configuration file

[sources.http]
type = "http_server"
address = "0.0.0.0:8080"
encoding = "json"
healthcheck = true


[sinks.aws_cloudwatch_logs]
type = "aws_cloudwatch_logs"
inputs = ["http"]
compression = "none"
create_missing_group = true
create_missing_stream = true
group_name = "test_opentelemetry"
region = "ap-south-1"
stream_name = "test"
encoding.codec = "json"

I am getting following error

nuc2kor@PUA-V-0001T:~$ sudo RUST_LOG=aws_smithy_http=trace vector -c /etc/vector/vector.toml 
2023-11-01T08:26:26.497705Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-11-01T08:26:26.499104Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-11-01T08:26:26.571570Z  INFO vector::topology::running: Running healthchecks.
2023-11-01T08:26:26.571650Z  INFO vector: Vector has started. debug="false" version="0.33.0" arch="x86_64" revision="89605fb 2023-09-27 14:18:24.180809939"
2023-11-01T08:26:26.571659Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-11-01T08:26:26.571872Z  INFO source{component_kind="source" component_id=http component_type=http_server component_name=http}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:8080
2023-11-01T08:26:27.573499Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: failed to construct request component_kind="sink" component_type="aws_cloudwatch_logs" component_id=aws_cloudwatch_logs component_name=aws_cloudwatch_logs
2023-11-01T08:26:45.710592Z  INFO sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=2
2023-11-01T08:26:46.711527Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: failed to construct request internal_log_rate_limit=true
2023-11-01T08:26:46.711549Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being suppressed to avoid flooding.
2023-11-01T08:26:46.711583Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ConstructionFailure(ConstructionFailure { source: SigningStageError { kind: MissingCredentials } }))) request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-11-01T08:26:46.711602Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=2 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

The sink I have configured is same in both the cases, but I am not able to send the data to the cloud watch using http)server or file as source.

Configuration

[sources.http]
type = "http_server"
address = "0.0.0.0:8080"
encoding = "json"
healthcheck = true


[sinks.aws_cloudwatch_logs]
type = "aws_cloudwatch_logs"
inputs = ["http"]
compression = "none"
create_missing_group = true
create_missing_stream = true
group_name = "test_opentelemetry"
region = "ap-south-1"
stream_name = "test"
encoding.codec = "json"

Version

0.33.0

Debug Output

nuc2kor@PUA-V-0001T:~$ sudo RUST_LOG=aws_smithy_http=trace vector -c /etc/vector/vector.toml 
2023-11-01T08:26:26.497705Z  INFO vector::app: Log level is enabled. level="vector=info,codec=info,vrl=info,file_source=info,tower_limit=info,rdkafka=info,buffers=info,lapin=info,kube=info"
2023-11-01T08:26:26.499104Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-11-01T08:26:26.571570Z  INFO vector::topology::running: Running healthchecks.
2023-11-01T08:26:26.571650Z  INFO vector: Vector has started. debug="false" version="0.33.0" arch="x86_64" revision="89605fb 2023-09-27 14:18:24.180809939"
2023-11-01T08:26:26.571659Z  INFO vector::app: API is disabled, enable by setting `api.enabled` to `true` and use commands like `vector top`.
2023-11-01T08:26:26.571872Z  INFO source{component_kind="source" component_id=http component_type=http_server component_name=http}: vector::sources::util::http::prelude: Building HTTP server. address=0.0.0.0:8080
2023-11-01T08:26:27.573499Z ERROR vector::topology::builder: msg="Healthcheck failed." error=DescribeLogGroups failed: failed to construct request component_kind="sink" component_type="aws_cloudwatch_logs" component_id=aws_cloudwatch_logs component_name=aws_cloudwatch_logs
2023-11-01T08:26:45.710592Z  INFO sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::aws_cloudwatch_logs::service: Sending events. events=2
2023-11-01T08:26:46.711527Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Non-retriable error; dropping the request. error=CloudwatchError::Describe: failed to construct request internal_log_rate_limit=true
2023-11-01T08:26:46.711549Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector::sinks::util::retries: Internal log [Non-retriable error; dropping the request.] is being suppressed to avoid flooding.
2023-11-01T08:26:46.711583Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::service: Service call failed. No retries or retries exhausted. error=Some(Describe(ConstructionFailure(ConstructionFailure { source: SigningStageError { kind: MissingCredentials } }))) request_id=1 error_type="request_failed" stage="sending" internal_log_rate_limit=true
2023-11-01T08:26:46.711602Z ERROR sink{component_kind="sink" component_id=aws_cloudwatch_logs component_type=aws_cloudwatch_logs component_name=aws_cloudwatch_logs}:request{request_id=1}: vector_common::internal_event::component_events_dropped: Events dropped intentional=false count=2 reason="Service call failed. No retries or retries exhausted." internal_log_rate_limit=true

Example Data

No response

Additional Context

No response

References

No response

[StephenWakely]

The error you are getting is MissingCredentials.

I strongly suspect that this is because the first example is being run as your user and the second example, which doesn't work, is being run with sudo. Likely the root user has a different environment and has different, or non existent, AWS credentials configured. See here for more details on the various ways aws credentials can be configured in the environment.