Skip to content

Commit 82efd56

Browse files
tehcastertorvalds
authored andcommitted
locking/local_lock: fix _Generic() matching of local_trylock_t
Michael Larabel reported [1] a nginx performance regression in v6.15-rc3 and bisected it to commit 51339d9 ("locking/local_lock, mm: replace localtry_ helpers with local_trylock_t type") The problem is the _Generic() usage with a default association that masks the fact that "local_trylock_t *" association is not being selected as expected. Replacing the default with the only other expected type "local_lock_t *" reveals the underlying problem: include/linux/local_lock_internal.h:174:26: error: ‘_Generic’ selector of type ‘__seg_gs local_lock_t *’ is not compatible with any association The local_locki's are part of __percpu structures and thus the __percpu attribute is needed to associate the type properly. Add the attribute and keep the default replaced to turn any further mismatches into compile errors. The failure to recognize local_try_lock_t in __local_lock_release() means that a local_trylock[_irqsave]() operation will set tl->acquired to 1 (there's no _Generic() part in the trylock code), but then local_unlock[_irqrestore]() will not set tl->acquired back to 0, so further trylock operations will always fail on the same cpu+lock, while non-trylock operations continue to work - a lockdep_assert() is also not being executed in the _Generic() part of local_lock() code. This means consume_stock() and refill_stock() operations will fail deterministically, resulting in taking the slow paths and worse performance. Fixes: 51339d9 ("locking/local_lock, mm: replace localtry_ helpers with local_trylock_t type") Reported-by: Michael Larabel <Michael@phoronix.com> Closes: https://www.phoronix.com/review/linux-615-nginx-regression/2 [1] Signed-off-by: Vlastimil Babka <vbabka@suse.cz> Acked-by: Alexei Starovoitov <ast@kernel.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
1 parent 0251ddb commit 82efd56

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

include/linux/local_lock_internal.h

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -102,11 +102,11 @@ do { \
102102
l = (local_lock_t *)this_cpu_ptr(lock); \
103103
tl = (local_trylock_t *)l; \
104104
_Generic((lock), \
105-
local_trylock_t *: ({ \
105+
__percpu local_trylock_t *: ({ \
106106
lockdep_assert(tl->acquired == 0); \
107107
WRITE_ONCE(tl->acquired, 1); \
108108
}), \
109-
default:(void)0); \
109+
__percpu local_lock_t *: (void)0); \
110110
local_lock_acquire(l); \
111111
} while (0)
112112

@@ -171,11 +171,11 @@ do { \
171171
tl = (local_trylock_t *)l; \
172172
local_lock_release(l); \
173173
_Generic((lock), \
174-
local_trylock_t *: ({ \
174+
__percpu local_trylock_t *: ({ \
175175
lockdep_assert(tl->acquired == 1); \
176176
WRITE_ONCE(tl->acquired, 0); \
177177
}), \
178-
default:(void)0); \
178+
__percpu local_lock_t *: (void)0); \
179179
} while (0)
180180

181181
#define __local_unlock(lock) \

0 commit comments

Comments
 (0)