CP-50537: TGroup library to manage the priority and classify xapi execution threads. #6076
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
GabrielBuica
force-pushed
the
private/dbuica/feature/perf/tgroups
branch
5 times, most recently
from
de4fdfe to
d2b9a9d
Compare
GabrielBuica
changed the title
|
|
GabrielBuica
commented
Nov 4, 2024
GabrielBuica
commented
Nov 4, 2024
GabrielBuica
commented
Nov 4, 2024
GabrielBuica
commented
Nov 4, 2024
GabrielBuica
force-pushed
the
private/dbuica/feature/perf/tgroups
branch
from
d2b9a9d to
a831ee4
Compare
psafont
reviewed
Nov 5, 2024
psafont
reviewed
Nov 6, 2024
mg12
approved these changes
Nov 7, 2024
psafont
approved these changes
Nov 7, 2024
GabrielBuica
force-pushed
the
private/dbuica/feature/perf/tgroups
branch
2 times, most recently
from
61b3a38 to
578622e
Compare
|
This PR has a conflict now. |
Creates a new library `Tgroup`, that abstracts and manages groups of execution threads in xapi. When xapi is under load, all the threads need to share a single cpu in dom0 because of ocaml runtime single-cpu restrictions. This library is meant to orchestrate the threads in different priority groups. Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
`set_cgroup` adds the functionality of adding the current thread in a cgroup based on its creator. Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
Add functionality of setting the tgroup based on a http header named `originator`. Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
Clears the `extra_headers` of `UDSTransport` instance when making a connection. Previously, this was done only when tracing was enabled inside the `with_tracecontext` method to avoid the header duplication when `make_connection` was used multiple times. Currently, there is not other use of `add_extra_headers` or other update to the `_extra_headers`, making it safe to clear it when we make a new connection. (`xmlrpclib.Transport` updates the `_extra_headers` attribute only inside `make_connection` method but we override this method with our own for `UDSTransport`.) Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
XenAPI.py now passes an additional originator header when making requests to xapi, if the "ORIGINATOR" env var is present. Sm_exec now passes an env var, "ORIGINATOR=SM". To classify the threads correctly, we first need to determine the requests originators. This commit makes it possibly to explicitly state the originators as headers. Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
For now, the thread executing `Xapi.server_init` and it's children are classified as External. The only excception are http requests that come through the smapi internally. If those contain the originator header with the value set as "sm", the thread executing the request will be classified as internal. This represents the first phase of classifing xapi threads as internal vs external. Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
Adds a configurable variable in `xapi_globs`, `tgroups_enabled` that is meant to ask a guard for tgroup classification of the threads. If the guard is `false` all Tgroups functionality should act as a no op. For instance, adding the line: tgroups-enabled = false will result in the thread classification being skipped. Signed-off-by: Gabriel Buica <danutgabriel.buica@cloud.com>
GabrielBuica
force-pushed
the
private/dbuica/feature/perf/tgroups
branch
from
578622e to
efaf3f0
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the follow up to #6020
The initial phase is to classify the threads between Internal and External.
BVT + BST: 207007 (Dev Run)