Skip to content
This repository was archived by the owner on Oct 23, 2023. It is now read-only.

Add Akka Stream Source, Upgrade Libraries
Compare
Choose a tag to compare
@markglh markglh released this 28 Dec 21:05
· 16 commits to master since this release
v0.6.5
e731cd6
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Verified
Learn about vigilant mode.

Changes

Consumer

  • Update KCL to 1.8.8 (#44)
    • Support timeouts and retries for getRecords calls.
      • Applications can now use an async retry for getRecords calls to Kinesis. In addition to setting the retryGetRecordsInSeconds timeout, the application must also provide a maxGetRecordsThreadPool size for concurrent requests.
    • Support configuring the graceful shutdown timeout for MultiLang Clients using shutdownGraceMillis. This adds support for setting the timeout that the Java process will wait for the MutliLang client to complete graceful shutdown.
    • Add Record pre-fetching
      • This will retrieve and queue additional records from Kinesis while the application is processing existing records.
      • Prefetching can be enabled by setting dataFetchingStrategy to PREFETCH_CACHED. Once enabled an additional fetching thread will be started to retrieve records from Kinesis. Retrieved records will be held in a queue until the application is ready to process them.
      • Logging of long running tasks can be enabled by setting the logWarningForTaskAfterMillis configuration property
  • Add an Akka Streams Source that can be used with streams. It is possible to create a Source from a ConsumerConf or directly from the consumer name that is defined in the configuration. Every message that is emitted to the stream is of type CommitableEvent[ConsumerEvent] and has to be committed explicitly downstream with a call to event.commit(). It is possible to map to a different type of CommittableEvent via the map and mapAsync functionality. A KinesisConsumer is used internally for the Kinesis.source. All rules described for the KinesisConsumer also apply for the stream Source. (#39)
  • Update ConsumerEvent to use ByteBuffer instead of String. #41
  • Fix issue where ConsumerWorker with related CheckpointWorker wasn't stopped on lease loss. In case of a lost lease, KCL now calls the shutdown method of the record processor. (#40)

Producer

  • Update KPL to 0.12.8 (#44)

Migration guide

ConsumerEvent now returns a ByteBuffer instead of a String. Use payloadAsString to maintain previous functionality.

Massive thanks to @aquamatthias for the contributions

Assets 2
Loading