Configuring per-sink acknowledgements with a reduce transform

[andrewma2]

Hi, I'm running vector with an http_server source. We have two pipelines:

1. http_server -> remap -> aws_s3 sink
2. The same http_server -> reduce transform to combine logs -> aws_kinesis_streams sink

We have enabled E2E acknowledgements because we don't want to lose logs if the aws_s3 sink fails. However, based off the documentation, we would also wait on the aws_kinesis_streams sink. This takes longer since the reduce transform waits to aggregate logs, and our upstream HTTP client is actually timing out with this.

We decided that we're ok without the ack from aws_kinesis_streams, and we set acknowledgements.enabled to false on the sink, but it seems like it's still waiting on the logs to make it past the reduce before sending.

Am I doing something wrong with setting acknowledgements? And if not, is there a way to work around this so we can have E2E acknowledgements without needing to wait on the reduce transform (if we don't care about ack's in the downstream sink)?

Repro

To make it simple / easily reproducible, I made a similar toy case

Config

{
	"acknowledgements": {
		"enabled": true
	},
	"sources": {
                "in": {
                        "type": "http_server",
                        "address": "0.0.0.0:80"
                }
        },
	"transforms": {
		"hold": {
			"type": "reduce",
			"inputs": ["in"],
			"end_every_period_ms": 10000
		}
	},
	"sinks": {
		"out": {
			"acknowledgements": {
				"enabled": false
			},
			"type": "console",
			"inputs": ["hold"],
			"encoding": {
				"codec": "json"
			}
		},
		"print": {
			"type": "console",
			"inputs": ["in"],
			"encoding": {
				"codec": "json"
			}
		}
	}
}

This config is a simple HTTP server and two sinks. One sink is connected to the source and immediately prints, whereas another goes to a reduce transform and waits 10s before getting printed..

Command

curl 0.0.0.0:80 -X POST -d '{}'

Result

Even though ack is off for out, we wait 10s before we get a response on the curl command. In fact, even with acknowledgements.enabled on both sinks, we still wait 10s. It's only when acknowledgements.enabled is turned off globally that we get an immediate response.

[pront]

Hi @andrewma2, based on your reduce config, the transform will keep reducing for 10 seconds before it emits any event. So when you run your pipeline, it should take at least 10 seconds for the first event to hit any sink. When acks are off, then you get a response indicating that the source received the data, but no guarantees for the sinks.

[andrewma2]

Thanks for the response @pront ! So even with ack's off on the sink, we still need to wait for the logs to hit the sink? Is there any way around this? Ideally we'd like to be able to have E2E ack's enabled on the path without the reduce, and then not care about any ack results on the path with the reduce so we don't have to wait that 10s

Additionally, in the toy example, I noted that setting both sinks to have acks off still causes us to wait on the reduce, so does that mean having the reduce in any way means we have to wait 10s to ack (unless we turn off E2E acks globally)?

[pront]

Apologies for the late reply (although I vaguely recall commenting on a similar thread recently)

So even with ack's off on the sink, we still need to wait for the logs to hit the sink?

No, but if you have a reduce transform that can buffer for up to X seconds, then that would stop tour events from hitting the sink. With acks off, the source should immediately decode and emit the event and after that it will not process that single event ever again.