kernel: cbs: adding support for the Constant Bandwidth Server (CBS) #83601

alexpaschoaletto · 2025-01-06T16:45:22Z

This PR introduces kernel support for the Constant Bandwidth Server (CBS).

A CBS is an extension of the Earliest Deadline First (EDF) scheduler already implemented (although with some limitations) in Linux and RTEMS that allows tasks to be executed virtually isolated from each other (time-wise, not memory-wise), in a way that if a task executes for longer than expected it doesn't interfere on the execution of the others. In other words, the CBS prevents that a task
misbehavior causes other tasks to miss their own deadlines.

In a nutshell, the CBS is a work-conserving wrapper for the tasks that automatically recalculates their deadlines when they exceed their configured timing constraints. If a task takes longer than expected, it can be preempted by awaiting tasks instead of jeopardizing their deadlines.

You want to use the CBS when:

You want an automated way of recalculating the task deadlines.
You have a task set running under EDF and there needs to be a scheduling guarantee, that is, a guarantee that no task will ever starve because a higher priority one is executing for too long.
You want to improve the response time of lower priority tasks without jeopardizing the guarantees of the higher priority tasks.
You want to decide the task priorities by defining their guaranteed CPU time (for example, task 1 needs 50%, task 2 needs 30% and task 3 needs 20%), like a round-robin on steroids. That is possible because the ratio of the CBS attributes (budget, period) is equivalent to the guaranteed bandwidth of the server. It should be noted that if that's the case, the sum of all tasks' utilization should necessarily be less or equal than 100%.

So, depending on what you need, the CBS can be seen as a safe way of serving tasks with no deadlines within a EDF taskset (like a workqueue, but better) or a fail-safe for existing EDF tasks that might overrun, and thus endanger other tasks's deadlines.

The most important implementation file to analyze is this one. The rest are docs, samples, and eventual/small kernel mods to enable the CBS to work accordingly.

For more details, feel free to read RFC #78000 and the docs wrote for this contribution. Or just ask, I'll be happy to provide an answer.

andyross

Some quick notes after a quick and incomplete read. The big worry here is performance: this is adding two function calls and a bunch of tests to the context switch path for one somewhat obscure feature, and that's sort of a non-starter. At the very least, this should emit code that does an absolute minimum of work (e.g. just check a boolean on the thread struct) before returning into the main switch path. Calling out into a separate stack frame is spending a ton of cycles we can't spare.

Does the problem really require tracking budget across cooperative switches anyway? Can't you do this with a timer a-la timeslicing? Seems like this is taking a ton of overhead for what would normally seem like a fairly coarse decision. Thread deadlines are usually on the order of a tick, not a cycle.

andyross · 2025-01-07T02:04:04Z

kernel/thread.c

@@ -630,6 +630,9 @@ char *z_setup_new_thread(struct k_thread *new_thread,
 #ifdef CONFIG_SCHED_DEADLINE
 	new_thread->base.prio_deadline = 0;
 #endif /* CONFIG_SCHED_DEADLINE */
+#ifdef CONFIG_CBS
+	new_thread->cbs = NULL;
+#endif /* CONFIG_CBS */


Not really the fault of this PR, but we have to clean up the mess this function has become. Originally it had to initialize all fields manually because it was felt that doing a memset() was expensive, but at this point I doubt that's true, and the maintenance burden for sure isn't worth the benefit (it's been a while, but historically we had a ton of bugs with incompletely initialized data here).

agreed, however I just went with the flow here.

andyross · 2025-01-07T02:07:55Z

include/zephyr/sched_server/cbs.h

+} cbs_arg_t;
+
+typedef struct {
+	struct k_timer *timer;


k_timer is intended to be an intrusive data structure, just put it here by value. IIRC message queues are not, though.

andyross · 2025-01-07T02:09:10Z

kernel/sched.c

@@ -835,8 +839,16 @@ struct k_thread *z_swap_next_thread(void)
 /* Just a wrapper around arch_current_thread_set(xxx) with tracing */
 static inline void set_current(struct k_thread *new_thread)
 {
+	#ifdef CONFIG_CBS
+	cbs_thread_switched_out(_current_cpu->current);


The API for this is arch_current_cpu() now; some platforms have optimized variants that don't require fetching the CPU pointer (which requires running in a non-migratable context).

Can you confirm this information? I'm applying the fixes but the latest main branch seems to have it no more. It looks like _current is being used instead.

andyross · 2025-01-07T02:09:40Z

kernel/thread.c

@@ -942,6 +945,7 @@ void z_thread_mark_switched_out(void)
 }
 #endif /* CONFIG_INSTRUMENT_THREAD_SWITCHING */

+


Needless noop whitespace change nitpick

sorry 🫠 removed.

andyross · 2025-01-07T02:57:25Z

subsys/sched_server/cbs/cbs.c

+	cbs_arg_t *args = (cbs_arg_t *)cbs_args;
+
+	cbs->thread = k_current_get();
+	cbs->thread->cbs = cbs;


I don't see a lot of value to having the thread<->cbs pointer cycle? Seems like everywhere you are dealing with it is in the context of the thread being instrumented, no?

well, cbs->thread is needed for when if the CBS thread demands a deadline updated (when a job is pushed to it, see function k_cbs_push_job and cbs_check_current_bandwidth it calls). I can't think of a better way of fetching the CBS thread if not like this.

The thread->cbs is needed for the context switches; the cbs timer should start when the CBS thread switches in and stop when it switches out. And again, I can't think of a better way of fetching the CBS thread if not like this.

I'm open to suggestions if you wish.

andyross · 2025-01-07T02:59:59Z

subsys/sched_server/cbs/cbs.c

+	}
+	cbs_cycle_t now = cbs_get_now();
+	cbs_t *cbs = (cbs_t *)k_timer_user_data_get(timer);
+	k_sched_lock();


The sched_lock facility isn't SMP-safe. You almost certainly want a spinlock here unless you are taking care to prevent contention from other CPUs.

ok-dokey, I have changed to K_SPINLOCK.

andyross · 2025-01-07T03:00:51Z

subsys/sched_server/cbs/cbs.c

+		return;
+	}
+	cbs_cycle_t now = cbs_get_now();
+	cbs_t *cbs = (cbs_t *)k_timer_user_data_get(timer);


FWIW if you make the k_timer a cbs field, you can just use a CONTAINER_OF() here to get the timer pointer, which is a single addition.

great suggestion! done.

andyross · 2025-01-07T03:01:39Z

include/zephyr/sched_server/cbs.h

+typedef struct {
+	cbs_callback_t function;
+	void *arg;
+} cbs_job_t;


No typedef struct names in zephyr please, absent special circumstances like by-value opaque types. Should just be "struct cbs_job"

ok, I've removed typedef structs.

andyross · 2025-01-07T03:03:01Z

subsys/sched_server/cbs/cbs.c

+
+void cbs_thread_switched_in(struct k_thread *thread)
+{
+	if (!thread || !thread->cbs) {


Why check thread for null here? That would be a bug in the context switch primitives you hooked. This is extremely performance-sensitive code.

Sorry, it's just me used to include defensive code by default. I have removed that.

andyross · 2025-01-07T03:03:55Z

subsys/sched_server/cbs/cbs.c

+		return;
+	}
+	cbs_t *cbs = (cbs_t *)thread->cbs;
+	if (cbs->is_initialized && !cbs->is_idle) {


Likewise no defensive coding in the context switch path please. If you want to be sure cbs is initialized then the architecture needs to be affirmatively sure it is before that pointer is set on the thread. It's OK to make this an assert if you're worried about design bugs.

It's ok, this code wasn't needed indeed. I have removed it.

nashif · 2025-01-11T17:01:13Z

quick note here, you should not be introducing a core scheduling/kernel feature as a new subsystem, this belongs into the already existing kernel "subsystem"

This commit introduces kernel support for the Constant Bandwidth Server (CBS), an extension of the Earliest Deadline First (EDF) scheduler that allows tasks to be executed virtually isolated from each other, in a way that if a task executes for longer than expected it doesn’t interfere on the execution of the others. In other words, the CBS prevents that a task misbehavior causes other tasks to miss their own deadlines. In a nutshell, the CBS is a work-conserving wrapper for the tasks that automatically recalculates their deadlines when they exceed their configured timing constraints, thus acting as a fail-safe for EDF overruns. Signed-off-by: Alexander pinheiro paschoaletto <1222703@isep.ipp.pt>

This commit simply adds the documentation for the CBS. Signed-off-by: Alexander pinheiro paschoaletto <1222703@isep.ipp.pt>

…CBS) This commit adds two sample codes for the the CBS, one for testing a very simple logging application and other for effectively implementing it alongside other hard real-time threads. Signed-off-by: Alexander pinheiro paschoaletto <1222703@isep.ipp.pt>

alexpaschoaletto · 2025-02-14T14:33:58Z

Some quick notes after a quick and incomplete read. The big worry here is performance: this is adding two function calls and a bunch of tests to the context switch path for one somewhat obscure feature, and that's sort of a non-starter. At the very least, this should emit code that does an absolute minimum of work (e.g. just check a boolean on the thread struct) before returning into the main switch path. Calling out into a separate stack frame is spending a ton of cycles we can't spare.

Thank you for your brief review, I have corrected most of the requests. FWIW, the functions called in the context switch are there to just start/stop the CBS timer if the thread belongs to a CBS. There is nothing else.

By the way, the absolute worst measured overheads in the XIAO ESP32C3 (RISC-V, single-core, 160MHz) were of 128 clock cycles for switching in and 139 clock cycles for switching out. These translate to a period in the magnitude of nanoseconds. The averages were (obviously) lower: 110.92 and 90.94 clock cycles, respectively. Does that looks inadmissible?

Does the problem really require tracking budget across cooperative switches anyway? Can't you do this with a timer a-la timeslicing? Seems like this is taking a ton of overhead for what would normally seem like a fairly coarse decision. Thread deadlines are usually on the order of a tick, not a cycle.

Yes it does. If thread X preempts CBS Y, the CBS needs a way of re-starting its budget timer when thread X finishes and it is allowed to resume execution. I couldn't make that work without including these functions in k_swap.

github-actions · 2025-04-16T00:37:56Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

alexpaschoaletto · 2025-04-16T17:06:34Z

I'd like the label removed, please.

Copilot

Pull Request Overview

This PR introduces kernel support for the Constant Bandwidth Server (CBS) to improve scheduling guarantees and isolate task execution time effects. Key changes include initializing a new CBS field in thread structures, adding CBS logging and internal implementations, and updating scheduler and swap routines to integrate CBS functionality.

Reviewed Changes

Copilot reviewed 24 out of 35 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
kernel/thread.c	Initializes the new CBS field in thread structures.
kernel/sched_server/cbs_log.c	Adds logging support for CBS events.
kernel/sched_server/cbs.c	Implements CBS functions including budget management and job handling.
kernel/sched.c & kernel/include/kswap.h	Updates scheduler routines to start/stop CBS timers during context switches.
Header files in kernel/include/zephyr/sched_server/	Provides public and internal API definitions for CBS.

Files not reviewed (11)

doc/introduction/index.rst: Language not supported
doc/kernel/services/index.rst: Language not supported
doc/kernel/services/sched_server/cbs.rst: Language not supported
doc/zephyr.doxyfile.in: Language not supported
kernel/CMakeLists.txt: Language not supported
kernel/Kconfig: Language not supported
kernel/Kconfig.sched_server: Language not supported
samples/cbs/cbs.rst: Language not supported
samples/cbs/edf/CMakeLists.txt: Language not supported
samples/cbs/edf/README.rst: Language not supported
samples/cbs/edf/prj.conf: Language not supported

Comments suppressed due to low confidence (1)

kernel/sched_server/cbs_log.c:21

Typo in log message: 'SWT_AY' should likely be 'SWT_AWAY' for consistency with similar log messages.

LOG_INF("%s	SWT_AY	%u", (char *)cbs->name, (uint32_t)cbs->budget.current);

github-actions · 2025-06-26T00:39:47Z

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

alexpaschoaletto · 2025-07-07T17:20:12Z

please

This pull request has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 14 days. Note, that you can always re-open a closed pull request at any time.

I won't lie, this is a bit frustrating. But yes, I'd like to have the stale label removed, please. I intend to include a little bit more functionality to this code in the near future.

cfriedt

Just a quick fly-by. I can review the actual scheduler code soon.

cfriedt · 2025-07-08T10:48:09Z

include/zephyr/kernel/thread.h

@@ -372,6 +372,11 @@ struct k_thread {
 	_wait_q_t  halt_queue;
 #endif /* CONFIG_SMP */

+#ifdef CONFIG_CBS
+	/** Constant Bandwidth Server (CBS) struct */
+	void *cbs;


It might be preferable to forward declare a struct and specifically type this field rather than using a void pointer.

cfriedt · 2025-07-08T10:49:43Z

include/zephyr/sched_server/cbs.h

+#ifndef ZEPHYR_CBS
+#define ZEPHYR_CBS


Can you use the full path of the function in header guards like in other header files? I realize it's a bit verbose, but it's conventional and unique.

cfriedt · 2025-07-08T10:52:48Z

include/zephyr/sched_server/cbs.h

+typedef uint32_t cbs_cycle_t;
+#endif
+
+struct cbs_job {


I would prefix all of these structs with k_ as well

cfriedt · 2025-07-08T10:53:54Z

include/zephyr/sched_server/cbs.h

+ */
+
+#ifdef CONFIG_TIMER_HAS_64BIT_CYCLE_COUNTER
+typedef uint64_t cbs_cycle_t;


I'm a little surprised we don't already have a cycle counter type like this.

can you prefix this with a k_ please?

cfriedt · 2025-07-08T10:57:50Z

include/zephyr/sched_server/cbs.h

+	k_timeout_t period;
+};
+
+struct k_cbs {


Is this supposed to be a special thread structure for CBS?

The main reason I ask is because Zephyr already has some Kconfig options or defines for specifying maximum thread name length and whether or not there is a field for thread name.

I guess the other reason is to see if there is a convention for naming used by other schedulers for similar types.

The name of the struct leaves me wondering and could maybe be expanded a bit to be more descriptive

cfriedt · 2025-07-08T10:59:50Z

include/zephyr/sched_server/cbs.h

+#endif
+};
+
+extern void cbs_thread(void *job_queue, void *cbs_struct, void *unused);


Also should be prefixed with k_.

Is it possible for the job queue use named struct?

can do the prefix thing, sure. But I don't know if I understood the second request, though. Do you want type void * to be changed to an explicit k_msgq * struct? Aint't thread entry points supposed to be formatted as having all arguments as void *?

cfriedt · 2025-07-08T11:04:23Z

kernel/Kconfig.sched_server

+			Scheduling delay (in milliseconds) for the CBS thread.
+
+	if CBS_LOG
+	config CBS_THREAD_MAX_NAME_LEN


This likely duplicates something that already exists in Zephyr, so I would suggest using the existing convention.

cfriedt · 2025-07-08T11:08:55Z

kernel/Kconfig.sched_server

+		requires CONFIG_CBS to be included within the project.
+
+if CBS_LOG
+menu "CBS events to be logged"


The events being recorded here really seem like they would fit better with Zephyr's tracing subsystem. Can they use that mechanism instead?

alexpaschoaletto · 2025-07-10T19:16:17Z

Just a quick fly-by. I can the actual scheduler code soon.

Thank you so much for taking your time, cfriedt! I'm currently busy with other stuff at the moment, but I will get back to this eventually and address all your comments.

zephyrbot added area: Documentation area: Kernel area: Samples Samples area: Documentation Infrastructure labels Jan 6, 2025

zephyrbot requested review from andyross, carlescufi, ceolin, cfriedt, dcpleung, gmarull, kartben, nashif, npitre, peter-mitsis and TaiJuWu January 6, 2025 16:46

zephyrbot assigned kartben Jan 6, 2025

alexpaschoaletto mentioned this pull request Jan 6, 2025

New Feature: Constant Bandwidth Server (CBS) #78000

Open

alexpaschoaletto force-pushed the cbs branch from 55f23a2 to 9e397c0 Compare January 6, 2025 21:20

kartben assigned andyross and peter-mitsis and unassigned kartben Jan 7, 2025

alexpaschoaletto force-pushed the cbs branch from 9e397c0 to 94e42ff Compare January 7, 2025 01:10

andyross reviewed Jan 7, 2025

View reviewed changes

alexpaschoaletto force-pushed the cbs branch 2 times, most recently from 3beedf7 to 6ce9413 Compare January 8, 2025 19:32

alexpaschoaletto force-pushed the cbs branch 4 times, most recently from 64eec3b to 6e1a25e Compare January 12, 2025 12:45

alexpaschoaletto force-pushed the cbs branch 6 times, most recently from 9bb4d30 to b9b6a25 Compare January 14, 2025 11:33

alexpaschoaletto force-pushed the cbs branch from b9b6a25 to 5a0d198 Compare January 20, 2025 16:44

Alexander pinheiro paschoaletto added 3 commits January 31, 2025 09:50

docs: CBS: adding docs for the Constant Bandwidth Server (CBS)

97650f1

This commit simply adds the documentation for the CBS. Signed-off-by: Alexander pinheiro paschoaletto <1222703@isep.ipp.pt>

alexpaschoaletto force-pushed the cbs branch from 5a0d198 to 533d43a Compare January 31, 2025 12:57

alexpaschoaletto requested a review from andyross February 14, 2025 14:34

github-actions bot added the Stale label Apr 16, 2025

kartben removed the Stale label Apr 16, 2025

nashif requested a review from Copilot April 26, 2025 15:17

Copilot AI reviewed Apr 26, 2025

View reviewed changes

github-actions bot added the Stale label Jun 26, 2025

github-actions bot removed the Stale label Jul 8, 2025

cfriedt reviewed Jul 8, 2025

View reviewed changes

		@@ -942,6 +945,7 @@ void z_thread_mark_switched_out(void)
		}
		#endif /* CONFIG_INSTRUMENT_THREAD_SWITCHING */

		#ifndef ZEPHYR_CBS
		#define ZEPHYR_CBS

kernel: cbs: adding support for the Constant Bandwidth Server (CBS) #83601

Are you sure you want to change the base?

kernel: cbs: adding support for the Constant Bandwidth Server (CBS) #83601

Conversation

alexpaschoaletto commented Jan 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andyross left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nashif commented Jan 11, 2025

Uh oh!

alexpaschoaletto commented Feb 14, 2025

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

alexpaschoaletto commented Apr 16, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

github-actions bot commented Jun 26, 2025

Uh oh!

alexpaschoaletto commented Jul 7, 2025

Uh oh!

cfriedt left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

alexpaschoaletto commented Jan 6, 2025 •

edited

Loading

cfriedt left a comment •

edited

Loading