Skip to content

Address feedback on Lamda page #9895

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Address feedback on Lamda page #9895

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
efc3245
Address feedback on Lamda page
Naarcha-AWS May 13, 2025
7249728
Merge branch 'main' into data-prepper-feedback
Naarcha-AWS May 14, 2025
2f0a15e
Apply suggestions from code review
Naarcha-AWS May 22, 2025
770e93d
Apply suggestions from code review
Naarcha-AWS Jun 12, 2025
13f9237
Update aws-lambda.md
Naarcha-AWS Jun 25, 2025
f297671
Apply suggestions from code review
Naarcha-AWS Jul 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 18 additions & 21 deletions _data-prepper/pipelines/configuration/processors/aws-lambda.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,60 +10,57 @@

The [AWS Lambda](https://aws.amazon.com/lambda/) integration allows developers to use serverless computing capabilities within their OpenSearch Data Prepper pipelines for flexible event processing and data routing.

## AWS Lambda processor configuration

The `aws_lambda` processor enables invocation of an AWS Lambda function within your Data Prepper pipeline in order to process events. It supports both synchronous and asynchronous invocations based on your use case.

## Configuration fields

You can configure the processor using the following configuration options.

Field | Type | Required | Description
-------------------- | ------- | -------- | ----------------------------------------------------------------------------
Field | Type | Required | Description
:--- | :------- | :--- | :---
`function_name` | String | Required | The name of the AWS Lambda function to invoke.
`invocation_type` | String | Required | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`.
`invocation_type` | String | Required | Specifies the [invocation type](#invocation-type), either `request-response` or `event`. Default is `request-response`.
`aws.region` | String | Required | The AWS Region in which the Lambda function is located.
`aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function.
`max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`.
`batch` | Object | Optional | The batch settings for the Lambda invocations. Default is `key_name = "events"`. Default threshold is `event_count=100`, `maximum_size="5mb"`, and `event_collect_timeout = 10s`.
`lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor.
`response_codec` | Object | Optional | A codec configuration for parsing Lambda responses. Default is `json`.
`tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error.
`sdk_timeout` | Duration| Optional | Configures the SDK's client connection timeout period. Default is `60s`.
`connection_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`.

Check failure on line 31 in _data-prepper/pipelines/configuration/processors/aws-lambda.md

View workflow job for this annotation

GitHub Actions / vale

[vale] _data-prepper/pipelines/configuration/processors/aws-lambda.md#L31 

[OpenSearch.Spelling] Error: SDK's. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.
Raw output 
{"message": "[OpenSearch.Spelling] Error: SDK's. If you are referencing a setting, variable, format, function, or repository, surround it with tic marks.", "location": {"path": "_data-prepper/pipelines/configuration/processors/aws-lambda.md", "range": {"start": {"line": 31, "column": 68}}}, "severity": "ERROR"}
`response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`.

#### Example configuration
The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this field in the table above if it is not supported? Are there discrepancies between these docs and what AWS supports? If so are they documented anywhere? In my mind adding just this blurb adds more confusion.

Copy link
Contributor Author

@Naarcha-AWS Naarcha-AWS Jun 3, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rpash: Based on your experience (issue #9872), that would seem to be the case. That being said, I have no additional insight on what additional discrepancies may exist between Data Prepper, OSI, and what AWS supports. This PR is only meant to address feedback on the AWS Lamda page specifically.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dlvenable or @srikanthjg: Is there an alternative method or parameter for adjusting the invocation_type? Should we just remove the field and section from the docs entirely?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for opensource dataprepper, this field is needed an is used in both lambda processor and sink.

  1. request-response (default for the processor): This is mapped to AWS Lambda's
    "RequestResponse" invocation type. When using this mode. This is synchronous processing, where the pipeline waits for the Lambda result

  2. event (default for the sink): This is mapped to AWS Lambda's "Event" invocation type.
    When using this mode The request is sent to Lambda asynchronously and Data Prepper doesn't wait for a response

{: .warning}


## Usage

This example configures the `aws_lambda` processor to invoke a Lambda function with advanced batching, conditional invocation, retry logic, and cross-region AWS credentials:

```
processors:
- aws_lambda:
function_name: "my-lambda-function"
invocation_type: "request-response"
response_events_match: false
aws:
function_name: "arn:aws:lambda:us-west-2:123456789012:function:my-processing-function"
batch:
threshold:
event_collect_timeout: 30s
maximum_size: "5mb"
aws:
region: "us-east-1"
sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role"
max_retries: 3
batch:
key_name: "events"
threshold:
event_count: 100
maximum_size: "5mb"
event_collect_timeout: PT10S
lambda_when: "event['status'] == 'process'"

```
{% include copy-curl.html %}

## Usage
## Invocation types

The processor supports the following invocation types:

- `request-response`: The processor waits for Lambda function completion before proceeding.
- `event`: The function is triggered asynchronously without waiting for a response.
- `batch`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout.
- `codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs.
- `tags_on_match_failure`: Custom tags can be applied to events when Lambda processing fails or encounters unexpected issues.

## Behavior

Expand Down
Loading