Skip to content

Honor Vertex Read-Batch-Size independent of ISB partition count #2895

@vigith

Description

@vigith

Currently read-batch-size at pod level (at a vertex) is computed as

partition-read-batch = read-batch-size

pod-read-batch = read-batch-size * partition count

i.e., if we set 300 as batch-size and if we have 3 partitions, the pod cummulative read batch size is 900.

The above is wrong because vertex setting for a pod is inadvertently overridden by changing partition count.

Fix

Reach Batch Size for a pod at partition level should be inversely proportional to ISB partition count

partition-read-batch = read-batch-size / partition count

pod-read-batch = read-batch-size

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingdocumentationImprovements or additions to documentation

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions