Skip to content

[WIP] Concurrent read large file #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

LuQQiu
Copy link
Collaborator

@LuQQiu LuQQiu commented Jul 3, 2023

Prerequisite: Alluxio REST API return file size from list operation (preferred) or get file status operation
Current issue: Alluxio REST API perf may bottlenecked by Alluxio server thread pool size

@LuQQiu
Copy link
Collaborator Author

LuQQiu commented Jul 3, 2023

have an assumption that Alluxio server is bottlenecked (only have limited threads executing the I/O tasks)
Alluxio http server workergroup is EventLoopGroup workerGroup = new NioEventLoopGroup(); This line creates a group with a default number of threads for handling the actual I/O work (reading and writing data). The default number of threads is 2 * number of available processors, as per Netty’s documentation.
DEFAULT_EVENT_LOOP_THREADS = Math.max(1, SystemPropertyUtil.getInt(
“io.netty.eventLoopThreads”, NettyRuntime.availableProcessors() * 2));
available processors may not show actual value in container env from Shawn and my experience

Alluxio/alluxio#17021

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant