Subscriber

Overview

AresDB subscriber consumes messages from Kafka; transforms data based on configuration; upserts batch to sink (AresDB server or Kafka). It is designed as message lossless and supports AresDB single node mode and distributed mode.

AresDB Single Node Mode

In AresDB single node mode, subscriber periodically fetches job assignments from controller, consumes messages from Kafka, transforms and batches messages, finally directly sends batch of rows to one AresDB server.

AresDB Distributed Mode

In AresDB distributed mode, subscriber periodically fetches job assignments and enum dictionary from controller, consumes messages from Kafka, transforms messages to AresDB upsert batch format, partitions upsert batches based on table primary keys, finally publishes batches to its specified partitionID in Kafka. On AresDB cluster side, each server consumes upsert batches for the assigned shards.