From efc3245a1b58046a2a393c31632ebf0fc75ecf74 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 13 May 2025 15:02:05 -0500 Subject: [PATCH 1/5] Address feedback on Lamda page Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuration/processors/aws-lambda.md | 26 +++++++++++-------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index 0ef9dfd7d74..67706de2d28 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -10,7 +10,6 @@ nav_order: 10 The [AWS Lambda](https://aws.amazon.com/lambda/) integration allows developers to use serverless computing capabilities within their OpenSearch Data Prepper pipelines for flexible event processing and data routing. -## AWS Lambda processor configuration The `aws_lambda` processor enables invocation of an AWS Lambda function within your Data Prepper pipeline in order to process events. It supports both synchronous and asynchronous invocations based on your use case. @@ -18,8 +17,8 @@ The `aws_lambda` processor enables invocation of an AWS Lambda function within y You can configure the processor using the following configuration options. -Field | Type | Required | Description --------------------- | ------- | -------- | ---------------------------------------------------------------------------- +Field | Type | Required | Description +:--- | :------- | :--- | :--- `function_name` | String | Required | The name of the AWS Lambda function to invoke. `invocation_type` | String | Required | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`. `aws.region` | String | Required | The AWS Region in which the Lambda function is located. @@ -29,18 +28,24 @@ Field | Type | Required | Description `lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor. `response_codec` | Object | Optional | A codec configuration for parsing Lambda responses. Default is `json`. `tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error. -`sdk_timeout` | Duration| Optional | Configures the SDK's client connection timeout period. Default is `60s`. +`sdk_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. -#### Example configuration +The invocation_type field is not supported in Amazon OpenSearch Ingestion pipelines using the aws_lambda processor. Including this field will result in a validation error. +{: .note} + + +## Usage + +This example configures the `aws_lambda` processor to invoke a Lambda function with advanced batching, conditional invocation, retry logic, and cross-region AWS credentials: ``` processors: - aws_lambda: - function_name: "my-lambda-function" - invocation_type: "request-response" - response_events_match: false - aws: + arn: "arn:aws:lambda:us-west-2:123456789012:function:my-processing-function" + timeout: 2000 + batch_size: 10 + aws: region: "us-east-1" sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role" max_retries: 3 @@ -51,11 +56,10 @@ processors: maximum_size: "5mb" event_collect_timeout: PT10S lambda_when: "event['status'] == 'process'" - ``` {% include copy-curl.html %} -## Usage +## Invocation types The processor supports the following invocation types: From 2f0a15e41b4ec703ed6590da1dfc0880bf7b1252 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 22 May 2025 16:16:37 -0500 Subject: [PATCH 2/5] Apply suggestions from code review Co-authored-by: David Venable Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _data-prepper/pipelines/configuration/processors/aws-lambda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index 67706de2d28..bda874d2351 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -31,7 +31,7 @@ Field | Type | Required | Description `sdk_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. -The invocation_type field is not supported in Amazon OpenSearch Ingestion pipelines using the aws_lambda processor. Including this field will result in a validation error. +The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error. {: .note} From 770e93db9bf35a86177960582547dd3f7b77e918 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 12 Jun 2025 18:17:09 -0500 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: David Venable Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../pipelines/configuration/processors/aws-lambda.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index bda874d2351..c195bc82503 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -28,7 +28,7 @@ Field | Type | Required | Description `lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor. `response_codec` | Object | Optional | A codec configuration for parsing Lambda responses. Default is `json`. `tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error. -`sdk_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. +`connection_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error. @@ -42,9 +42,11 @@ This example configures the `aws_lambda` processor to invoke a Lambda function w ``` processors: - aws_lambda: - arn: "arn:aws:lambda:us-west-2:123456789012:function:my-processing-function" - timeout: 2000 - batch_size: 10 + function_name: "arn:aws:lambda:us-west-2:123456789012:function:my-processing-function" + batch: + threshold: + event_collect_timeout: 30s + event_count: 10 aws: region: "us-east-1" sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role" From 13f92379aa18af8e54561b45439cf399be891d0d Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Wed, 25 Jun 2025 13:45:08 -0500 Subject: [PATCH 4/5] Update aws-lambda.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuration/processors/aws-lambda.md | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index c195bc82503..c5b7330c042 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -20,7 +20,7 @@ You can configure the processor using the following configuration options. Field | Type | Required | Description :--- | :------- | :--- | :--- `function_name` | String | Required | The name of the AWS Lambda function to invoke. -`invocation_type` | String | Required | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`. +`invocation_type` | String | Required | Specifies the [invocation type](#invocation-type), either `request-response` or `event`. Default is `request-response`. `aws.region` | String | Required | The AWS Region in which the Lambda function is located. `aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function. `max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`. @@ -32,7 +32,7 @@ Field | Type | Required | Description `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error. -{: .note} +{: .warning} ## Usage @@ -46,17 +46,11 @@ processors: batch: threshold: event_collect_timeout: 30s - event_count: 10 + maximum_size: "5mb" aws: region: "us-east-1" sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role" max_retries: 3 - batch: - key_name: "events" - threshold: - event_count: 100 - maximum_size: "5mb" - event_collect_timeout: PT10S lambda_when: "event['status'] == 'process'" ``` {% include copy-curl.html %} @@ -67,9 +61,6 @@ The processor supports the following invocation types: - `request-response`: The processor waits for Lambda function completion before proceeding. - `event`: The function is triggered asynchronously without waiting for a response. -- `batch`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout. -- `codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs. -- `tags_on_match_failure`: Custom tags can be applied to events when Lambda processing fails or encounters unexpected issues. ## Behavior From f297671dc627b4dbc3d2e55f94472af8a3d7e3bb Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 8 Jul 2025 06:20:28 -0500 Subject: [PATCH 5/5] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _data-prepper/pipelines/configuration/processors/aws-lambda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index c5b7330c042..6b0dc246462 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -20,7 +20,7 @@ You can configure the processor using the following configuration options. Field | Type | Required | Description :--- | :------- | :--- | :--- `function_name` | String | Required | The name of the AWS Lambda function to invoke. -`invocation_type` | String | Required | Specifies the [invocation type](#invocation-type), either `request-response` or `event`. Default is `request-response`. +`invocation_type` | String | Required | Specifies the [invocation type](#invocation-types), either `request-response` or `event`. Default is `request-response`. `aws.region` | String | Required | The AWS Region in which the Lambda function is located. `aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function. `max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`.