Skip to content

A buffering worker #155

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AlexHill opened this issue Mar 17, 2025 · 0 comments
Open

A buffering worker #155

AlexHill opened this issue Mar 17, 2025 · 0 comments

Comments

@AlexHill
Copy link

I have a use-case that the existing concepts don't quite seem to cover, and I'm wondering how I might extend ProcessScheduler to support it.

What I'd like is a kind of combination of a cumulative worker and a concurrent buffer, that works with SelectWorkers. The worker should have an associated buffer of a given size, and each task will use a certain amount of that buffer. A task may be assigned to a worker if the unused buffer is greater than the amount the task requires. The buffer level is automatically increased and decreased when a task starts and ends.

An applicable analogy is a CPU (worker) with access to a certain amount of RAM (buffer) running concurrent processes that each require a certain amount of RAM.

With a small patch, I can nearly fake this using the existing API:

  • for each worker, create a CumulativeWorker and a ConcurrentBuffer
  • set the buffer's initial capacity to its upper bound, and its lower bound to zero
  • for each task that needs this worker, both unload and load the worker's buffer (this is where the patch was necessary)

This appears to do exactly what I want. But it can't be used with SelectWorkers, because the buffers can't be selectively used. I need to somehow express the constraint that a task should only load and unload the buffer of the workers it is assigned to.

Can you give me any pointers as to where I might go from here?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant