Skip to content

Commit 0bb11a3

Browse files
paulmckrcufbq
authored andcommitted
rcu-tasks: Maintain real-time response in rcu_tasks_postscan()
The current code will scan the entirety of each per-CPU list of exiting tasks in ->rtp_exit_list with interrupts disabled. This is normally just fine, because each CPU typically won't have very many tasks in this state. However, if a large number of tasks block late in do_exit(), these lists could be arbitrarily long. Low probability, perhaps, but it really could happen. This commit therefore occasionally re-enables interrupts while traversing these lists, inserting a dummy element to hold the current place in the list. In kernels built with CONFIG_PREEMPT_RT=y, this re-enabling happens after each list element is processed, otherwise every one-to-two jiffies. [ paulmck: Apply Frederic Weisbecker feedback. ] Link: https://lore.kernel.org/all/ZdeI_-RfdLR8jlsm@localhost.localdomain/ Signed-off-by: Paul E. McKenney <paulmck@kernel.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Sebastian Siewior <bigeasy@linutronix.de> Cc: Anna-Maria Behnsen <anna-maria@linutronix.de> Cc: Steven Rostedt <rostedt@goodmis.org> Signed-off-by: Boqun Feng <boqun.feng@gmail.com>
1 parent 1612160 commit 0bb11a3

File tree

1 file changed

+21
-1
lines changed

1 file changed

+21
-1
lines changed

kernel/rcu/tasks.h

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -968,13 +968,33 @@ static void rcu_tasks_postscan(struct list_head *hop)
968968
*/
969969

970970
for_each_possible_cpu(cpu) {
971+
unsigned long j = jiffies + 1;
971972
struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rcu_tasks.rtpcpu, cpu);
972973
struct task_struct *t;
974+
struct task_struct *t1;
975+
struct list_head tmp;
973976

974977
raw_spin_lock_irq_rcu_node(rtpcp);
975-
list_for_each_entry(t, &rtpcp->rtp_exit_list, rcu_tasks_exit_list)
978+
list_for_each_entry_safe(t, t1, &rtpcp->rtp_exit_list, rcu_tasks_exit_list) {
976979
if (list_empty(&t->rcu_tasks_holdout_list))
977980
rcu_tasks_pertask(t, hop);
981+
982+
// RT kernels need frequent pauses, otherwise
983+
// pause at least once per pair of jiffies.
984+
if (!IS_ENABLED(CONFIG_PREEMPT_RT) && time_before(jiffies, j))
985+
continue;
986+
987+
// Keep our place in the list while pausing.
988+
// Nothing else traverses this list, so adding a
989+
// bare list_head is OK.
990+
list_add(&tmp, &t->rcu_tasks_exit_list);
991+
raw_spin_unlock_irq_rcu_node(rtpcp);
992+
cond_resched(); // For CONFIG_PREEMPT=n kernels
993+
raw_spin_lock_irq_rcu_node(rtpcp);
994+
t1 = list_entry(tmp.next, struct task_struct, rcu_tasks_exit_list);
995+
list_del(&tmp);
996+
j = jiffies + 1;
997+
}
978998
raw_spin_unlock_irq_rcu_node(rtpcp);
979999
}
9801000

0 commit comments

Comments
 (0)