Non-Streamy KCL Consumer Offering

[etspaceman]

Hello -

This discussion is a bit weird as, naturally, we would not consider a non-FS2 offering in something literally named fs2-aws. But I think there's value to this, as this library largely serves as the typelevel compliant KCL Consumer in the community.

I propose that we build a non-streamy KCL Consumer, which still uses the ChunkedRecordProcessor. The reasons for this are as follows:

• The overhead of maintaining the records received in a global queue across all ShardRecordProcessor instances is removed
• The KCL is designed to be concurrent via the ShardRecordProcessor itself, i.e. we have 1 processor per shard consumed. The FS2 approach effectively uses that concurrency layer to provide the FS2 Stream itself, so that users can fine-tune their applications in an FS2 way. That's great! But I've found in my testing of this approach is that there is a cost in terms of how well the concurrency matches up.
• Checkpointing can be handled within the offering itself, so users do not have to consider it in their applications

For users that care about avoiding that overhead, a non-streamy offering is valuable. Such as:

trait KinesisResource[F[_]] {

  def readFromKinesisStream(appName: String, streamName: String)(cb: Chunk[CommittableRecord] => Unit): Resource[F, Unit] =
    readFromKinesisStream(KinesisConsumerSettings(streamName, appName)(cb))

  def readFromKinesisStream(consumerConfig: KinesisConsumerSettings)(cb: Chunk[CommittableRecord] => Unit): Resource[F, Unit]

}

This also is related to #976, as KinesisConsumerSettings could be isolated from the FS2 queue settings.

Wanted to start a discussion and gather thoughts from the group.

Thanks all!

[semenodm]

Can you elaborate on your proposal here? As far as I understand, instead o to enquiring message to the global fs2 queue, you suggest what exactly?

[etspaceman]

Sure. Sorry, I realize my initial proposal needed to be edited to include the callback function from the ChunkedRecordProcessor. I've edited it above.

Effectively, a user just provides the settings + that callback, and runs it as a resource in the background. So instead of the callback enqueing to a queue, it runs a function defined by the user, and then checkpoints once the function is complete.