The bandwidth of the read file benchmark doesn't scale up when -test.cpu becomes large

[xin3liang]

The bandwidth of the read file benchmark doesn't scale up when -test.cpu becomes large.
See below test results, for test BenchmarkGoFuseMemoryRead, the bandwidth doesn't scale up when -test.cpu is 32+.
And for test BenchmarkGoFuseFDRead, the bandwidth doesn't scale up when -test.cpu is ~2+.

[root@client5 go-fuse]# go test ./benchmark -test.bench 'Read$' -test.cpu 1,2,4,8,16,32,64,$(nproc)
goos: linux
goarch: amd64
pkg: github.com/hanwen/go-fuse/v2/benchmark
cpu: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
BenchmarkGoFuseMemoryRead                   3536            348005 ns/op        6026.22 MB/s        3573 B/op        103 allocs/op
BenchmarkGoFuseMemoryRead-2                 5542            230751 ns/op        9088.37 MB/s        4011 B/op        111 allocs/op
BenchmarkGoFuseMemoryRead-4                 7869            177888 ns/op        11789.16 MB/s       2897 B/op        102 allocs/op
BenchmarkGoFuseMemoryRead-8                 7573            146180 ns/op        14346.37 MB/s       3010 B/op         97 allocs/op
BenchmarkGoFuseMemoryRead-16               13089             88321 ns/op        23744.54 MB/s       2914 B/op         96 allocs/op
BenchmarkGoFuseMemoryRead-32               15603             74119 ns/op        28294.26 MB/s       3472 B/op         96 allocs/op
BenchmarkGoFuseMemoryRead-64               16168             74109 ns/op        28298.19 MB/s       3669 B/op         96 allocs/op
BenchmarkGoFuseMemoryRead-112              15554             89165 ns/op        23519.89 MB/s       4365 B/op         96 allocs/op
BenchmarkGoFuseFDRead                      18832             61235 ns/op        1070.23 MB/s         122 B/op          3 allocs/op
BenchmarkGoFuseFDRead-2                    34098             43279 ns/op        1514.26 MB/s          95 B/op          2 allocs/op
BenchmarkGoFuseFDRead-4                    29010             36657 ns/op        1787.82 MB/s          92 B/op          2 allocs/op
BenchmarkGoFuseFDRead-8                    28642             42491 ns/op        1542.35 MB/s         107 B/op          2 allocs/op
BenchmarkGoFuseFDRead-16                   28728             39150 ns/op        1673.96 MB/s         116 B/op          2 allocs/op
BenchmarkGoFuseFDRead-32                   32385             40201 ns/op        1630.20 MB/s         112 B/op          2 allocs/op
BenchmarkGoFuseFDRead-64                   35319             38679 ns/op        1694.36 MB/s         108 B/op          2 allocs/op
BenchmarkGoFuseFDRead-112                  28196             41337 ns/op        1585.41 MB/s         113 B/op          2 allocs/op
PASS
ok      github.com/hanwen/go-fuse/v2/benchmark  44.812s

[xin3liang]

It looks like the readers int is hardcoded to 32 or 1:
https://github.com/hanwen/go-fuse/blob/master/benchmark/read_test.go#L23

[hanwen]

the FDRead benchmark is there for verifying that the backing store works (try running as root).

The number of readers is large because I don't want the test to be bottlenecked by the number of readers.

Note that go-fuse is unlikely to work well with high numbers of CPUs. The request reading/dispatch loops have no affinity to CPUs and request reading synchronizes across CPUs as well.

[xin3liang]

the FDRead benchmark is there for verifying that the backing store works (try running as root).

The number of readers is large because I don't want the test to be bottlenecked by the number of readers.

Note that go-fuse is unlikely to work well with high numbers of CPUs. The request reading/dispatch loops have no affinity to CPUs and request reading synchronizes across CPUs as well.

Thanks for the explanation, @hanwen.
You are right. I find that the kernel fuse locking overhead is heavy for 32 CPUs, as shown in the perf top and flame graph below.

test.cpu=32
Samples: 1M of event 'cycles', 4000 Hz, Event count (approx.): 747156085684 lost: 0/0 drop: 0/0
Overhead  Shared Object                        Symbol
  41.22%  [kernel]                             [k] native_queued_spin_lock_slowpath
  14.48%  [kernel]                             [k] memcpy_erms
  11.88%  [kernel]                             [k] _raw_spin_lock
   1.28%  [kernel]                             [k] try_grab_compound_head
   1.22%  [kernel]                             [k] gup_pte_range
   1.00%  [kernel]                             [k] syscall_return_via_sysret
   0.91%  [kernel]                             [k] unlock_request.part.0
   0.91%  [kernel]                             [k] fuse_copy_page
   0.91%  benchmark.test                       [.] 0x000000000007aea4

And regarding CPU affinity, because the kernel fuse shares the same resources(e.g., request queues), for the single-job FIO test, I do find that CPU core binding benefits the bandwidth performance. But for the multi-job FIO throughput test, I find that it's hard to do the CPU core binding across many CPU cores.

[xin3liang]

BTW, through the go trace.out, I also found that go-fuse creates too many request reading/dispatch loops.
For BenchmarkGoFuseMemoryRead test (test.cpu=1 and -test.benchtime=1s), loops' count is 10088.
For BenchmarkGoFuseMemoryRead test (test.cpu=1 and -test.benchtime=30s), loops' count is 337112.

Not sure if it is a problem. And libfuse only creates a small number of loops(max_threads default to 10):

		if (mt->numavail == 0 && mt->numworker < mt->max_threads)
			fuse_loop_start_thread(mt);

See:
https://github.com/libfuse/libfuse/blob/master/lib/fuse_loop_mt.c#L174
https://github.com/libfuse/libfuse/blob/master/lib/fuse_loop_mt.c#L476

Pages show that too many goroutines will impact the performance:
https://medium.com/@karam.majdi33/why-your-goroutines-are-slower-than-you-think-433372e6e50b
https://medium.com/@_orcaman/when-too-much-concurrency-slows-you-down-golang-9c144ca305a

[xin3liang]

Another finding is that increasing the blockSize from 64k to 128k(the default fuse request size) gets a better bandwidth for the read benchmark tests.

[root@client5 go-fuse]# go test ./benchmark -test.bench 'Read$' -test.cpu 1,2,4,8,16,32,64,$(nproc)
goos: linux
goarch: amd64
pkg: github.com/hanwen/go-fuse/v2/benchmark
cpu: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
BenchmarkGoFuseMemoryRead                   2151            535546 ns/op        7831.83 MB/s        4336 B/op        103 allocs/op
BenchmarkGoFuseMemoryRead-2                 3291            371947 ns/op        11276.61 MB/s       4491 B/op        104 allocs/op
BenchmarkGoFuseMemoryRead-4                 4821            278265 ns/op        15073.08 MB/s       4166 B/op        102 allocs/op
BenchmarkGoFuseMemoryRead-8                 6928            183566 ns/op        22849.01 MB/s       3493 B/op         97 allocs/op
BenchmarkGoFuseMemoryRead-16               12121             99850 ns/op        42006.14 MB/s       3311 B/op         96 allocs/op
BenchmarkGoFuseMemoryRead-32               14302             93033 ns/op        45083.97 MB/s       3418 B/op         96 allocs/op
BenchmarkGoFuseMemoryRead-64               13545             79256 ns/op        52920.65 MB/s       4735 B/op         96 allocs/op
BenchmarkGoFuseMemoryRead-112              14826             80977 ns/op        51796.03 MB/s       4477 B/op         96 allocs/op
BenchmarkGoFuseFDRead                      13725             96549 ns/op        1357.57 MB/s         233 B/op          5 allocs/op
BenchmarkGoFuseFDRead-2                    17103             74482 ns/op        1759.79 MB/s         144 B/op          4 allocs/op
BenchmarkGoFuseFDRead-4                    17438             71752 ns/op        1826.73 MB/s         150 B/op          4 allocs/op
BenchmarkGoFuseFDRead-8                    16225             71256 ns/op        1839.44 MB/s         179 B/op          4 allocs/op
BenchmarkGoFuseFDRead-16                   15928             71492 ns/op        1833.37 MB/s         198 B/op          4 allocs/op
BenchmarkGoFuseFDRead-32                   16738             60115 ns/op        2180.35 MB/s         218 B/op          4 allocs/op
BenchmarkGoFuseFDRead-64                   16582             70625 ns/op        1855.88 MB/s         195 B/op          4 allocs/op
BenchmarkGoFuseFDRead-112                  17972             71861 ns/op        1823.95 MB/s         194 B/op          4 allocs/op
PASS
ok      github.com/hanwen/go-fuse/v2/benchmark  53.296s

[xin3liang]

FYI, here is the patch that can scale BenchmarkGoFuseFDRead's bandwidth as -test.cpu increasing.
In case someone needs it.

go-fuse-benchmark.patch

[root@client5 go-fuse]# go test ./benchmark -test.bench 'Read$' -test.cpu 1,2,4,8,16,32,64,$(nproc)
goos: linux
goarch: amd64
pkg: github.com/hanwen/go-fuse/v2/benchmark
cpu: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
BenchmarkGoFuseMemoryRead                  29002             35385 ns/op        3704.22 MB/s          76 B/op          3 allocs/op
BenchmarkGoFuseMemoryRead-2                33342             39295 ns/op        6671.12 MB/s         200 B/op          7 allocs/op
BenchmarkGoFuseMemoryRead-4                19042             60068 ns/op        8728.21 MB/s         430 B/op         12 allocs/op
BenchmarkGoFuseMemoryRead-8                21412             56937 ns/op        18416.51 MB/s        745 B/op         24 allocs/op
BenchmarkGoFuseMemoryRead-16               16629             65484 ns/op        32025.46 MB/s       1745 B/op         48 allocs/op
BenchmarkGoFuseMemoryRead-32               15051             79188 ns/op        52966.70 MB/s       3623 B/op         96 allocs/op
BenchmarkGoFuseMemoryRead-64                7437            169915 ns/op        49369.30 MB/s      10605 B/op        196 allocs/op
BenchmarkGoFuseMemoryRead-112               4244            262586 ns/op        55905.69 MB/s      18013 B/op        354 allocs/op
BenchmarkGoFuseFDRead                      11181             90156 ns/op        1453.84 MB/s         259 B/op          5 allocs/op
BenchmarkGoFuseFDRead-2                    15040             81810 ns/op        3204.30 MB/s         394 B/op          6 allocs/op
BenchmarkGoFuseFDRead-4                    13215             97760 ns/op        5362.99 MB/s         653 B/op          7 allocs/op
BenchmarkGoFuseFDRead-8                    11167            107332 ns/op        9769.43 MB/s        1175 B/op         11 allocs/op
BenchmarkGoFuseFDRead-16                    8271            160396 ns/op        13074.81 MB/s       3373 B/op         19 allocs/op
BenchmarkGoFuseFDRead-32                    5191            221976 ns/op        18895.32 MB/s       6186 B/op         35 allocs/op
BenchmarkGoFuseFDRead-64                    3020            369075 ns/op        22728.76 MB/s      10632 B/op         69 allocs/op
BenchmarkGoFuseFDRead-112                   1986            808257 ns/op        18162.62 MB/s      18876 B/op        120 allocs/op
PASS
ok      github.com/hanwen/go-fuse/v2/benchmark  43.858s

[hanwen]

Thank you for the patch. I think that makes sense; I've applied it, https://review.gerrithub.io/c/hanwen/go-fuse/+/1215792

[hanwen]

BTW, through the go trace.out, I also found that go-fuse creates too many request reading/dispatch loops.
For BenchmarkGoFuseMemoryRead test (test.cpu=1 and -test.benchtime=1s), loops' count is 10088.
For BenchmarkGoFuseMemoryRead test (test.cpu=1 and -test.benchtime=30s), loops' count is 337112.

Not sure if it is a problem. And libfuse only creates a small number of loops(max_threads default to 10):

what happens if you try 11 concurrent accesses on a networked file system, and there is network slowness? Would't that mean there is no thread left to service requests?

We create a loop to make sure there is always something trying to read the device. it's possible this can be better, but it's hard to say without first making a benchmark that measures what "better" means.

[xin3liang]

BTW, through the go trace.out, I also found that go-fuse creates too many request reading/dispatch loops.
For BenchmarkGoFuseMemoryRead test (test.cpu=1 and -test.benchtime=1s), loops' count is 10088.
For BenchmarkGoFuseMemoryRead test (test.cpu=1 and -test.benchtime=30s), loops' count is 337112.

Not sure if it is a problem. And libfuse only creates a small number of loops(max_threads default to 10):

what happens if you try 11 concurrent accesses on a networked file system, and there is network slowness? Would't that mean there is no thread left to service requests?

My current tests are on Juicefs single-node, no networking now.

We create a loop to make sure there is always something trying to read the device. it's possible this can be better, but it's hard to say without first making a benchmark that measures what "better" means.

Let's move this topic to another issue. I draft a patch and do some benchmarks, and post the test results on that issue:
#565 (comment)
So we can close this issue, as the scale-up patch is merged.