Skip to content
This repository was archived by the owner on Jun 3, 2021. It is now read-only.

Conversation

BenoitHanotte
Copy link

@BenoitHanotte BenoitHanotte commented Mar 19, 2019

This is a work in progress to throttle the hive split discovery.

If the value of hive.max-outstanding-splits is greater than 0, then HiveSplitSource will use the ThrottledAsyncQueue to enqueue the splits, which throttles the returned batches to release at most that amount of splits per seconds to the consumer, thus throttling the split discovery mechanism for hive.

Note that hive.max-outstanding-splits needs to be more than query.min-schedule-split-batch-size otherwise BufferingSplitSource will buffer the throttled splits and assign them all at once, cancelling the "smoothing" effect of throttling the splits.

@BenoitHanotte BenoitHanotte changed the base branch from master to criteo-master March 19, 2019 13:16
@BenoitHanotte BenoitHanotte force-pushed the DPM-144 branch 6 times, most recently from b0c0294 to 769a78f Compare March 21, 2019 17:33
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant