Skip to content

feat: Add support for SQS events in Amazon.Lambda.Annotations #1758

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
Jun 17, 2024
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions Libraries/src/Amazon.Lambda.Annotations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -729,8 +729,8 @@ The `SQSEvent` attribute contains the following properties:
* **Queue** (Required) - The SQS queue that will act as the event trigger for the Lambda function. This can either be the queue ARN or reference to the SQS queue resource that is already defined in the serverless template. To reference a SQS queue resource in the serverless template, prefix the resource name with "@" symbol.
* **ResourceName** (Optional) - The CloudFormation resource name for the SQS event source mapping. By default this is set to the SQS queue name if `Queue` is set to an SQS queue ARN. If `Queue` is set to an existing CloudFormation resource, than that is used as the default value without the "@" prefix.
* **Enabled** (Optional) - If set to false, the event source mapping will be disabled and message polling will be paused. Default value is true.
* **BatchSize** (Optional) - The maximum number of messages that will be sent for processing in a single batch. This value must be between 1 to 10000. Default value is 10.
* **MaximumBatchingWindowInSeconds** (Optional) - The maximum amount of time, in seconds, to gather records before invoking the function. This value must be between 0 to 300. Default value is 0.
* **BatchSize** (Optional) - The maximum number of messages that will be sent for processing in a single batch. This value must be between 1 to 10000. For FIFO queues the maximum allowed value is 10. Default value is 10.
* **MaximumBatchingWindowInSeconds** (Optional) - The maximum amount of time, in seconds, to gather records before invoking the function. This value must be between 0 to 300. Default value is 0. When BatchSize is set to a value greater than 10 MaximumBatchingWindowInSeconds must be set to at least 1. This property must not be set if the event source mapping is being created for a FIFO queue.
* **Filters** (Optional) - A collection of semicolon (;) separated strings where each string denotes a pattern. Only those SQS messages that conform to at least 1 pattern will be forwarded to the Lambda function for processing.
* **MaximumConcurrency** (Optional) - The maximum number of concurrent Lambda invocations that the SQS queue can trigger. This value must be between 2 to 1000. The default value is 1000.

Expand All @@ -741,8 +741,7 @@ The Lambda method must conform to the following rules when it is tagged with the
1. It must have at least 1 argument and can have at most 2 arguments.
- The first argument is required and must be of type `SQSEvent` defined in the [Amazon.Lambda.SQSEvents](https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.SQSEvents/SQSEvent.cs) package.
- The second argument is optional and must be of type `ILambdaContext` defined in the [Amazon.Lambda.Core](https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.Core/ILambdaContext.cs) package.
2. The method return type must be one of `void`, `Task`, `SQSBatchResponse` or `Task<SQSBatchResponse>`. The `SQSBatchResponse` type is defined in the [Amazon.Lambda.SQSEvents](https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.SQSEvents/SQSBatchResponse.cs) package. If the return type is `SQSBatchResponse` or `Task<SQSBatchResponse>`, then the [FunctionResponseTypes](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html#sam-function-sqs-functionresponsetypes) in the event source mapping is set to
report `ReportBatchItemFailures`
2. The method return type must be one of `void`, `Task`, `SQSBatchResponse` or `Task<SQSBatchResponse>`. The `SQSBatchResponse` type is defined in the [Amazon.Lambda.SQSEvents](https://github.com/aws/aws-lambda-dotnet/blob/master/Libraries/src/Amazon.Lambda.SQSEvents/SQSBatchResponse.cs) package. If the return type is `SQSBatchResponse` or `Task<SQSBatchResponse>`, then the [FunctionResponseTypes](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-property-function-sqs.html#sam-function-sqs-functionresponsetypes) in the event source mapping is set to report `ReportBatchItemFailures`

```csharp

Expand All @@ -759,7 +758,7 @@ In the above example `TestQueue` refers to an existing SQS queue resource in the
The following SQS event source mapping will be generated for the `SQSMessageHandler` Lambda function

```json
"SQSMessageHandler": {
"SQSMessageHandler": {
"Type": "AWS::Serverless::Function",
"Metadata": {
"Tool": "Amazon.Lambda.Annotations",
Expand Down
26 changes: 24 additions & 2 deletions Libraries/src/Amazon.Lambda.Annotations/SQS/SQSEventAttribute.cs
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ public bool Enabled

/// <summary>
/// The maximum number of messages that will be sent for processing in a single batch.
/// This value must be between 1 to 10000. Default value is 10.
/// This value must be between 1 to 10000. For FIFO queues the maximum allowed value is 10. Default value is 10.
/// </summary>
public uint BatchSize
{
Expand All @@ -80,7 +80,9 @@ public uint BatchSize

/// <summary>
/// The maximum amount of time, in seconds, to gather records before invoking the function.
/// This value must be between 0 to 300. Default value is 0.
/// This value must be between 0 to 300. Default value is 0.
/// When <see cref="BatchSize"/> is set to a value greater than 10 <see cref="MaximumBatchingWindowInSeconds"/> must be set to at least 1.
/// This property must not be set if the event source mapping is being created for a FIFO queue.
/// </summary>
public uint MaximumBatchingWindowInSeconds
{
Expand Down Expand Up @@ -134,6 +136,26 @@ internal List<string> Validate()
{
validationErrors.Add($"{nameof(SQSEventAttribute.MaximumBatchingWindowInSeconds)} = {MaximumBatchingWindowInSeconds}. It must be between 0 and 300");
}
if (IsBatchSizeSet && BatchSize > 10 && (!IsMaximumBatchingWindowInSecondsSet || MaximumBatchingWindowInSeconds < 1))
{
validationErrors.Add($"{nameof(SQSEventAttribute.MaximumBatchingWindowInSeconds)} is not set or set to a value less than 1. " +
$"It must be set to atleast 1 when {nameof(SQSEventAttribute.BatchSize)} is greater than 10");
}

// The queue is FIFO if the queue ARN ends in ".fifo"
var isFifo = !Queue.StartsWith("@") && Queue.EndsWith(".fifo");
if (isFifo)
{
if (IsMaximumBatchingWindowInSecondsSet)
{
validationErrors.Add($"{nameof(SQSEventAttribute.MaximumBatchingWindowInSeconds)} must not be set when the event source mapping is for a FIFO queue");
}
if (IsBatchSizeSet && BatchSize > 10)
{
validationErrors.Add($"{nameof(SQSEventAttribute.BatchSize)} = {BatchSize}. It must be less than or equal to 10 when the event source mapping is for a FIFO queue");
}
}

if (!Queue.StartsWith("@"))
{
var arnTokens = Queue.Split(new char[] { ':' }, 6);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
}
]
},
"MaximumBatchingWindowInSeconds": 2,
"ScalingConfig": {
"MaximumConcurrency": 30
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1262,6 +1262,22 @@ public async Task VerifyInvalidSQSEvents_ThrowsCompilationErrors()
DiagnosticResult.CompilerError("AWSLambda0116")
.WithSpan($"TestServerlessApp{Path.DirectorySeparatorChar}SQSEventExamples{Path.DirectorySeparatorChar}InvalidSQSEvents.cs", 59, 9, 64, 10)
.WithArguments("ResourceName = . It must only contain alphanumeric characters and must not be an empty string"),

DiagnosticResult.CompilerError("AWSLambda0116")
.WithSpan($"TestServerlessApp{Path.DirectorySeparatorChar}SQSEventExamples{Path.DirectorySeparatorChar}InvalidSQSEvents.cs", 66, 9, 71, 10)
.WithArguments("MaximumBatchingWindowInSeconds is not set or set to a value less than 1. It must be set to atleast 1 when BatchSize is greater than 10"),

DiagnosticResult.CompilerError("AWSLambda0116")
.WithSpan($"TestServerlessApp{Path.DirectorySeparatorChar}SQSEventExamples{Path.DirectorySeparatorChar}InvalidSQSEvents.cs", 73, 9, 78, 10)
.WithArguments("MaximumBatchingWindowInSeconds is not set or set to a value less than 1. It must be set to atleast 1 when BatchSize is greater than 10"),

DiagnosticResult.CompilerError("AWSLambda0116")
.WithSpan($"TestServerlessApp{Path.DirectorySeparatorChar}SQSEventExamples{Path.DirectorySeparatorChar}InvalidSQSEvents.cs", 80, 9, 85, 10)
.WithArguments("BatchSize = 100. It must be less than or equal to 10 when the event source mapping is for a FIFO queue"),

DiagnosticResult.CompilerError("AWSLambda0116")
.WithSpan($"TestServerlessApp{Path.DirectorySeparatorChar}SQSEventExamples{Path.DirectorySeparatorChar}InvalidSQSEvents.cs", 80, 9, 85, 10)
.WithArguments("MaximumBatchingWindowInSeconds must not be set when the event source mapping is for a FIFO queue")
}
}
}.RunAsync();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,5 +62,26 @@ namespace TestServerlessApp.SQSEventExamples
{
Console.WriteLine($"Event processed: {evnt}");
}

[LambdaFunction]
[SQSEvent("@testQueue", BatchSize = 100)]
public void ProcessMessageWithMaximumBatchingWindowInSecondsNotSpecified(SQSEvent evnt)
{
Console.WriteLine($"Event processed: {evnt}");
}

[LambdaFunction]
[SQSEvent("@testQueue", BatchSize = 100, MaximumBatchingWindowInSeconds = 0)]
public void ProcessMessageWithMaximumBatchingWindowInSecondsLessThanOne(SQSEvent evnt)
{
Console.WriteLine($"Event processed: {evnt}");
}

[LambdaFunction]
[SQSEvent("arn:aws:sqs:us-east-2:444455556666:test-queue.fifo", BatchSize = 100, MaximumBatchingWindowInSeconds = 5)]
public void ProcessMessageWithFifoQueue(SQSEvent evnt)
{
Console.WriteLine($"Event processed: {evnt}");
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ namespace TestServerlessApp.SQSEventExamples
public class ValidSQSEvents
{
[LambdaFunction]
[SQSEvent("arn:aws:sqs:us-east-2:444455556666:queue1", BatchSize = 50, MaximumConcurrency = 30, Filters = "My-Filter-1; My-Filter-2")]
[SQSEvent("arn:aws:sqs:us-east-2:444455556666:queue1", BatchSize = 50, MaximumBatchingWindowInSeconds = 2, MaximumConcurrency = 30, Filters = "My-Filter-1; My-Filter-2")]
[SQSEvent("arn:aws:sqs:us-east-2:444455556666:queue2", MaximumBatchingWindowInSeconds = 5, Enabled = false)]
[SQSEvent("arn:aws:sqs:us-east-2:444455556666:my-queue")]
[SQSEvent("@testQueue", ResourceName = "testQueueEvent")]
Expand Down
Loading