Skip to content

Commit 7864431

Browse files
marquizmarquiz
committed
contrib: add example for enabling per-container RDT monitoring
Signed-off-by: Markus Lehtonen <markus.lehtonen@intel.com>
1 parent 10ffc61 commit 7864431

File tree

4 files changed

+216
-0
lines changed

4 files changed

+216
-0
lines changed

‎contrib/kustomize/samples/rdt-monitoring/README.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
# Per-container RDT monitoring with NRI hook-injector
2+
3+
This example demonstrates how to enable per-container RDT monitoring using OCI
4+
hooks. It works as a bridge-gap solution until native support
5+
(`linux.intelRdt.enableMonitoring` of OCI runtime-spec) is available.
6+
7+
References:
8+
9+
- https://github.com/opencontainers/runtime-spec/blob/main/config-linux.md#intelrdt
10+
- https://github.com/opencontainers/runc/pull/4832
11+
- https://github.com/containerd/nri/pull/215
12+
13+
## Design
14+
15+
The sample leverages the [hook-injector](../../../../plugins/hook-injector)
16+
sample plugin to inject OCI hooks that manage the per-container monitoring
17+
groups. It deploys a DaemonSet and consists of the following parts:
18+
19+
1. A custom location on the host (/etc/containers/nri/rdt-hook) is used to
20+
store the OCI hook binary and configuration.
21+
2. An init container creates the OCI hook binary on the host.
22+
3. A second init container creates the OCI hook configuration on the host.
23+
4. The hook-injector NRI plugin is run in the main container. The hook injector
24+
is configured to only watch for OCI hooks in the custom location.
25+
5. The hook is injected into all containers created after this.
26+
27+
> [!NOTE] The setup enables RDT monitoring for all new containers on the node.
28+
> The hook is injected into every container and the hook binary does not
29+
> provide any means to skip creation of the per-container monitoring group. In
30+
> most scenarios this shouldn't be a problem as the number of available
31+
> monitoring groups is high (in the order of hundreds).
32+
33+
## Deployment
34+
35+
Deploy the sample kustomize overlay:
36+
37+
```bash
38+
kubectl apply -k https://github.com/containerd/nri/contrib/kustomize/samples/rdt-monitoring
39+
```
40+
41+
The functionality can be verified by creating a new pod and checking the
42+
`/sys/fs/resctrl/<container-id>` directory on the node where the pod is
43+
deployed. An example:
44+
45+
```bash
46+
$ kubectl run --image registry.k8s.io/pause rdt-test
47+
48+
$ kubectl get pod rdt-test -o go-template='{{.spec.nodeName}}{{"\n"}}{{(index .status.containerStatuses 0).containerID}}{{"\n"}}'
49+
node-1
50+
containerd://477ba96b86e0f7756790d5c52ffa94feab0028f0785a23d143fd9390f09e35f6
51+
```
52+
53+
Then check the node:
54+
55+
```bash
56+
$ ssh node-1
57+
58+
$ cat /sys/fs/resctrl/mon_groups/477ba96b86e0f7756790d5c52ffa94feab0028f0785a23d143fd9390f09e35f6/tasks
59+
1817117
60+
1817149
61+
1817150
62+
1817151
63+
```

‎contrib/kustomize/samples/rdt-monitoring/initcontainers-patch.yaml‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,119 @@
1+
apiVersion: apps/v1
2+
kind: DaemonSet
3+
metadata:
4+
name: nri-plugin-hook-injector
5+
spec:
6+
template:
7+
spec:
8+
initContainers:
9+
- name: deploy-hook-binary
10+
image: busybox:latest
11+
command:
12+
- sh
13+
- -c
14+
- |
15+
echo "Creating OCI hook binary"
16+
cat > "$HOOKS_DIR/hook.sh" <<'EOF'
17+
#!/bin/sh
18+
fatal() {
19+
echo $@ >&2
20+
exit 1
21+
}
22+
23+
state=$(</dev/stdin)
24+
25+
resctrl_root_path="$(awk '$3 == "resctrl" {print $2; exit}' /proc/mounts)"
26+
if [ "$resctrl_root_path" = "" ]; then
27+
echo "resctrl mount point not found"
28+
exit 0
29+
fi
30+
31+
id=$(echo $state | jq -er .id) || fatal "failed to get container id"
32+
bundle=$(echo $state | jq -er .bundle) || fatal "failed to get container bundle path"
33+
34+
clos=""
35+
intel_rdt_config=$(jq -r .linux.intelRdt "$bundle/config.json")
36+
if [ "$intel_rdt_config" != "null" ]; then
37+
# If linux.intelRdt is non-null the container is assigned to a clos by the runtime
38+
clos=$(echo "$intel_rdt_config" | jq -er .closID) || clos="$id"
39+
fi
40+
41+
mon_group_path="$resctrl_root_path/$clos/mon_groups/$id"
42+
43+
create() {
44+
pid=$(echo $state | jq -er .pid) || fatal "failed to get container pid"
45+
46+
echo "creating monintoring group $mon_group_path"
47+
mkdir "$mon_group_path" || fatal "failed to create monitoring group for container $id"
48+
echo writing $pid to $mon_group_path/tasks
49+
50+
if ! echo $pid > "$mon_group_path/tasks"; then
51+
rmdir "$mon_group_path" || echo "failed to remove $mon_group_path" >&2
52+
fatal "failed to assign pid $pid to $mon_group_path"
53+
fi
54+
}
55+
56+
delete() {
57+
# MON group is reaped as part of the CLOS (by the runtime) if it was under
58+
# a dedicated CLOS created for this container
59+
if [ "$closid" != "$id" ]; then
60+
echo "deleting monintoring group $mon_group_path"
61+
rmdir "$mon_group_path" || fatal "failed to delete monitoring group of container $id"
62+
fi
63+
}
64+
65+
case "$1" in
66+
create)
67+
create
68+
;;
69+
delete)
70+
delete
71+
;;
72+
auto)
73+
status=$(echo $state | jq -er .status) || fatal "failed to get container status"
74+
case "$status" in
75+
creating)
76+
create
77+
;;
78+
stopped)
79+
delete
80+
;;
81+
esac
82+
;;
83+
*) fatal "unknown operation '$1'"
84+
esac
85+
EOF
86+
chmod +x "$HOOKS_DIR/hook.sh"
87+
env:
88+
- name: HOOKS_DIR
89+
value: /hooks
90+
volumeMounts:
91+
- name: etc-hooks-d
92+
mountPath: /hooks
93+
94+
- name: deploy-hook-config
95+
image: busybox:latest
96+
command:
97+
- sh
98+
- -c
99+
- |
100+
echo "Creating OCI hook config"
101+
cat > "$HOOKS_DIR/hook.json" <<'EOF'
102+
{
103+
"version": "1.0.0",
104+
"hook": {
105+
"path": "/etc/containers/nri/rdt-hook/hook.sh",
106+
"args": ["", "auto"]
107+
},
108+
"when": {
109+
"always": true
110+
},
111+
"stages": ["createRuntime", "poststop"]
112+
}
113+
EOF
114+
env:
115+
- name: HOOKS_DIR
116+
value: /hooks
117+
volumeMounts:
118+
- name: etc-hooks-d
119+
mountPath: /hooks

‎contrib/kustomize/samples/rdt-monitoring/kustomization.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: kustomize.config.k8s.io/v1beta1
2+
kind: Kustomization
3+
resources:
4+
- ../../hook-injector/unstable
5+
patches:
6+
- path: initcontainers-patch.yaml
7+
- path: volumes-patch.yaml

‎contrib/kustomize/samples/rdt-monitoring/volumes-patch.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
apiVersion: apps/v1
2+
kind: DaemonSet
3+
metadata:
4+
name: nri-plugin-hook-injector
5+
spec:
6+
template:
7+
spec:
8+
containers:
9+
- name: plugin
10+
volumeMounts:
11+
- name: etc-hooks-d
12+
mountPath: /etc/containers/nri/rdt-hook
13+
# Remove unwanted mounts
14+
- $patch: delete
15+
mountPath: /usr/share/containers/oci/hooks.d
16+
- $patch: delete
17+
mountPath: /usr/libexec/oci/hooks.d
18+
volumes:
19+
# Replace the default hooks directory with a custom one
20+
- name: etc-hooks-d
21+
hostPath:
22+
path: /etc/containers/nri/rdt-hook
23+
type: DirectoryOrCreate
24+
- $patch: delete
25+
name: usr-share-hooks-d
26+
- $patch: delete
27+
name: libexec-hooks-d

0 commit comments

Comments
 (0)