add internal log event for S3 sink

[marnao]

We'd like to see an internal log event whenever a file is written to S3 using the aws_s3 sink. This would be helpful for kicking off post processing actions when a file is written as well as troubleshooting. It would be great to have a record of when specific files were written. Right now it's kind of a black box. I am aware that AWS can do this natively with S3 events, but this would be for environments that are S3 compatible, not S3 native (think on-premises storage platforms).

We'd potentially be willing to contribute a pull request if this is a workable idea and there's interest in it. I'd prefer not to maintain a fork. I'm not sure if it would be as easy as the implementation in the aws_s3 source?

Let me know what you think!

Thanks.

[pront]

Hi @marnao,

I'm not sure if it would be as easy as the implementation in the aws_s3 source?

Yes, it is as easy as that. Did you consider making it a metric instead?
https://vector.dev/docs/reference/configuration/sinks/aws_s3/#telemetry

[marnao]

Thanks for the reply @pront !

I'm not sure if a metric makes sense here. I'm not just interested in the number of files being written but the actual path to those files. If we were to put those in a label the cardinality could be quite high depending on the volume of files going to the bucket. Unless I'm missing something I think a log event makes more sense. Ideally at the INFO level.

Assuming we go the log route, my only remaining question is whether it makes more sense to put it in the aws_s3 sink or s3_common? I am not sure what all uses s3_common and whether or not we'd want to propagate this to other places. My instinct says the aws_s3 sink is the appropriate spot, but open to suggestions.

[pront]

Sounds like an S3 specific log. We would put it in common if it's relevant for all users of common.