Skip to content

Commit 801a573

Browse files
committed
mm/percpu: micro-optimize pcpu_is_populated()
bitmap_next_clear_region() calls find_next_zero_bit() and find_next_bit() sequentially to find a range of clear bits. In case of pcpu_is_populated() there's a chance to return earlier if bitmap has all bits set. Signed-off-by: Yury Norov <yury.norov@gmail.com> Tested-by: Wolfram Sang <wsa+renesas@sang-engineering.com> Acked-by: Dennis Zhou <dennis@kernel.org>
1 parent 749443d commit 801a573

File tree

1 file changed

+8
-7
lines changed

1 file changed

+8
-7
lines changed

mm/percpu.c

Lines changed: 8 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1070,17 +1070,18 @@ static void pcpu_block_update_hint_free(struct pcpu_chunk *chunk, int bit_off,
10701070
static bool pcpu_is_populated(struct pcpu_chunk *chunk, int bit_off, int bits,
10711071
int *next_off)
10721072
{
1073-
unsigned int page_start, page_end, rs, re;
1073+
unsigned int start, end;
10741074

1075-
page_start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE);
1076-
page_end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE);
1075+
start = PFN_DOWN(bit_off * PCPU_MIN_ALLOC_SIZE);
1076+
end = PFN_UP((bit_off + bits) * PCPU_MIN_ALLOC_SIZE);
10771077

1078-
rs = page_start;
1079-
bitmap_next_clear_region(chunk->populated, &rs, &re, page_end);
1080-
if (rs >= page_end)
1078+
start = find_next_zero_bit(chunk->populated, end, start);
1079+
if (start >= end)
10811080
return true;
10821081

1083-
*next_off = re * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE;
1082+
end = find_next_bit(chunk->populated, end, start + 1);
1083+
1084+
*next_off = end * PAGE_SIZE / PCPU_MIN_ALLOC_SIZE;
10841085
return false;
10851086
}
10861087

0 commit comments

Comments
 (0)