From 143cfe6470b0434a722ef27e1144a8b8fd5848dc Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Wed, 16 Apr 2025 03:01:44 -0700 Subject: [PATCH 1/2] Updating container guidance for diag tools - Removed info about installation which is already covered in individual tool docs and has been unchanged for years - Add considerations about resource limits, security permissions, container size, and attack surface - Add information about the newer --diagnostic-port CLI option which doesn't require process namespace or /tmp sharing - Add warnings that tools (dotnet-dump especially) could cause containers with insufficient memory to be killed. - Warn that dotnet-dump writes out the dump relative to the target container file system view - Remove the weird recommendation that suggested perfcollect was only intended for pre-.NET Core 3.0 apps. - The text was unnecessarily refering to Docker. This guidance is likely to apply to any Linux container runtime, not just Docker. --- .../diagnostics/diagnostics-in-containers.md | 34 +++++++++++++------ 1 file changed, 23 insertions(+), 11 deletions(-) diff --git a/docs/core/diagnostics/diagnostics-in-containers.md b/docs/core/diagnostics/diagnostics-in-containers.md index 1c7143d3e5315..43ae02408b7d4 100644 --- a/docs/core/diagnostics/diagnostics-in-containers.md +++ b/docs/core/diagnostics/diagnostics-in-containers.md @@ -1,33 +1,45 @@ --- -title: Collect diagnostics in containers -description: Learn how .NET diagnostics tools for gathering performance traces and collecting dumps can be used in Docker containers. +title: Collect diagnostics in Linux containers +description: Learn how .NET diagnostics tools for gathering performance traces and collecting dumps can be used in Linux containers. ms.date: 09/01/2020 --- -# Collect diagnostics in containers +# Collect diagnostics in Linux containers -The same diagnostics tools that are useful for diagnosing .NET issues in other scenarios also work in Docker containers. However, some of the tools require special steps to work in a container. This article covers how tools for gathering performance traces and collecting dumps can be used in Docker containers. +The same diagnostics tools that are useful for diagnosing .NET issues in other scenarios also work in containers. The tools can be run in the same container as the target process, from the host, or from a sidecar container. ## Use .NET CLI tools in a container **These tools apply to: ✔️** .NET Core 3.1 SDK and later versions -The .NET global CLI diagnostic tools ([dotnet-counters](dotnet-counters.md), [dotnet-dump](dotnet-dump.md), [dotnet-gcdump](dotnet-gcdump.md), [dotnet-monitor](dotnet-monitor.md), and [dotnet-trace](dotnet-trace.md)) are designed to work in a wide variety of environments and should all work directly in Docker containers. Because of this, these tools are the preferred method of collecting diagnostic information for .NET scenarios targeting .NET Core 3.1 or later in containers. +All of the [dotnet-* CLI diagnostic tools](tools-overview.md#cli-tools) can work when run within the same container as the application they are inspecting but beware these potential trouble spots: -You can also install these tools without the .NET SDK by downloading the single-file variants from the links in the previous paragraph. These installs require a global install of the .NET runtime version 3.1 or later, which you can acquire following any of the prescribed methods in the [.NET installation documentation](../install/index.yml) or by consuming any of the official runtime containers. +- Tools run inside a container will be subject to container resource limits. The tools may run slowly or fail if the resource limits are too low. Most of the tools have fairly modest requirements but dotnet-dump and dotnet-gcdump can use considerable memory and disk space when targetting a process that has a large memory footprint. dotnet-trace and dotnet-counters might also create large files if they are configured to capture a large amount of trace events or metric time-series data. +- dotnet-dump will cause a helper process to run that needs ptrace permissions. Linux has numerous security configuration options that can affect whether this is successful so in some cases you may need to adjust the container's security configuration. See the [dump FAQ](faq-dumps.yml) for more guidance on diagnosing security privileges. +- When running tools inside the container you can either install them in advance when building the container or download them on-demand. Installing them in advance makes it easier when you need them but increases the size of the container and creates a larger attack surface that malicious actors could attempt to exploit. -### Use .NET global CLI tools in a sidecar container +## Use .NET CLI tools in a sidecar container or from the host -If you would like to use .NET global CLI diagnostic tools to diagnose processes in a different container, bear the following additional requirements in mind: +The [dotnet-* CLI diagnostic tools](tools-overview.md#cli-tools) also support running from the host or in a sidecar container. This largely avoids the size, security, and resource limitations of running in the same container but has some additional requirements for the tools to communicate successfully. -1. The containers must [share a process namespace](https://docs.docker.com/reference/cli/docker/container/run/#pid) (so that tools in the sidecar container can access processes in the target container). -2. The .NET global CLI diagnostic tools need access to files the .NET runtime writes to the /tmp directory, so the /tmp directory must be shared between the target and sidecar container via a volume mount. This could be done, for example, by having the containers share a common [volume](https://docs.docker.com/storage/volumes/#create-and-manage-volumes) or a Kubernetes [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume. If you attempt to use the diagnostic tools from a sidecar container without sharing the /tmp directory, you will get an error about the process "not running compatible .NET runtime." +When identifying a target process to inspect using the --process-id or --name tool command-line arguments, this requires: + +1. The containers must [share a process namespace](https://docs.docker.com/reference/cli/docker/container/run/#pid) so that tools in the sidecar container can access processes in the target container. +2. The tools need access to the [diagnostic port](diagnostic-port.md) Unix Domain Socket which the .NET runtime writes to the /tmp directory, so the /tmp directory must be shared between the target and sidecar container via a volume mount. This could be done, for example, by having the containers share a common [volume](https://docs.docker.com/storage/volumes/#create-and-manage-volumes) or a Kubernetes [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume. If you attempt to use the diagnostic tools from a sidecar container without sharing the /tmp directory, you will get an error about the process "not running compatible .NET runtime." + +If you don't want to share process namespace and the /tmp directory, many of the tools also support a more advanced --diagnostic-port command-line option. This allows you to directly specify the path to the target process' [diagnostic port](diagnostic-port.md) within the filesystem of the host or sidecar. The target container's /tmp directory will need to be mapped somewhere for this path to exist, but it doesn't have to be shared with the host or sidecar /tmp. + +> [!NOTE] +> Even when running the diagnostic tools from the host or sidecar, the target process may still be requested to do work which increases its resource usage inside the target container. We have observed dotnet-dump may cause the target process to page in substantial virtual memory when collecting a dump. Other tools may cause smaller impacts though we haven't seen these cause problems in practice. For example dotnet-trace requests the target process to allocate an event buffer and dotnet-counters requests a small memory region where metric data is aggregated. We recommend testing to ensure your memory limits aren't so tight that running the tools causes the OS to terminate your containers. + +> [!NOTE] +> When dotnet-dump writes a dump file to disk, the output path is interpretted in the context of the target process' view of the file system. This may differ from the host or sidecar container. ## Use `PerfCollect` in a container **This tool applies to: ✔️** .NET Core 2.1 and later versions -The [`PerfCollect`](./trace-perfcollect-lttng.md) script is useful for collecting performance traces and is the recommended tool for collecting traces prior to .NET Core 3.0. If using `PerfCollect` in a container, keep the following requirements in mind: +The [`PerfCollect`](./trace-perfcollect-lttng.md) script is useful for collecting performance traces that contain kernel events such as CPU samples or context switches. If using `PerfCollect` in a container, keep the following requirements in mind: - `PerfCollect` requires additional capabilities to run the `perf` tool. The minimal set of capabilities required is [`PERFMON`](https://man7.org/linux/man-pages/man7/capabilities.7.html) and [`SYS_PTRACE`](https://man7.org/linux/man-pages/man7/capabilities.7.html). Some environments require [`SYS_ADMIN`](https://man7.org/linux/man-pages/man7/capabilities.7.html). Be sure to start the container with [the necessary capabilities](https://docs.docker.com/engine/security/#linux-kernel-capabilities). If the minimal set doesn't work, then try with the full set. From f964827e013605cf850503f4ade9b3e156879c80 Mon Sep 17 00:00:00 2001 From: Noah Falk Date: Wed, 16 Apr 2025 16:44:52 -0700 Subject: [PATCH 2/2] Apply suggestions from code review Co-authored-by: Cam Soper --- .../diagnostics/diagnostics-in-containers.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/core/diagnostics/diagnostics-in-containers.md b/docs/core/diagnostics/diagnostics-in-containers.md index 43ae02408b7d4..3c60b1740cd52 100644 --- a/docs/core/diagnostics/diagnostics-in-containers.md +++ b/docs/core/diagnostics/diagnostics-in-containers.md @@ -1,7 +1,7 @@ --- title: Collect diagnostics in Linux containers description: Learn how .NET diagnostics tools for gathering performance traces and collecting dumps can be used in Linux containers. -ms.date: 09/01/2020 +ms.date: 04/16/2025 --- # Collect diagnostics in Linux containers @@ -14,26 +14,26 @@ The same diagnostics tools that are useful for diagnosing .NET issues in other s All of the [dotnet-* CLI diagnostic tools](tools-overview.md#cli-tools) can work when run within the same container as the application they are inspecting but beware these potential trouble spots: -- Tools run inside a container will be subject to container resource limits. The tools may run slowly or fail if the resource limits are too low. Most of the tools have fairly modest requirements but dotnet-dump and dotnet-gcdump can use considerable memory and disk space when targetting a process that has a large memory footprint. dotnet-trace and dotnet-counters might also create large files if they are configured to capture a large amount of trace events or metric time-series data. -- dotnet-dump will cause a helper process to run that needs ptrace permissions. Linux has numerous security configuration options that can affect whether this is successful so in some cases you may need to adjust the container's security configuration. See the [dump FAQ](faq-dumps.yml) for more guidance on diagnosing security privileges. +- Tools run inside a container will be subject to container resource limits. The tools may run slowly or fail if the resource limits are too low. Most of the tools have modest requirements but `dotnet-dump` and `dotnet-gcdump` can use considerable memory and disk space when targeting a process that has a large memory footprint. `dotnet-trace` and `dotnet-counters` might also create large files if they are configured to capture a large amount of trace events or metric time-series data. +- `dotnet-dump` will cause a helper process to run that needs ptrace permissions. Linux has numerous security configuration options that can affect whether this is successful so in some cases you may need to adjust the container's security configuration. See the [dump FAQ](faq-dumps.yml) for more guidance on diagnosing security privileges. - When running tools inside the container you can either install them in advance when building the container or download them on-demand. Installing them in advance makes it easier when you need them but increases the size of the container and creates a larger attack surface that malicious actors could attempt to exploit. ## Use .NET CLI tools in a sidecar container or from the host The [dotnet-* CLI diagnostic tools](tools-overview.md#cli-tools) also support running from the host or in a sidecar container. This largely avoids the size, security, and resource limitations of running in the same container but has some additional requirements for the tools to communicate successfully. -When identifying a target process to inspect using the --process-id or --name tool command-line arguments, this requires: +When identifying a target process to inspect using the `--process-id` or `--name` tool command-line arguments, this requires: 1. The containers must [share a process namespace](https://docs.docker.com/reference/cli/docker/container/run/#pid) so that tools in the sidecar container can access processes in the target container. -2. The tools need access to the [diagnostic port](diagnostic-port.md) Unix Domain Socket which the .NET runtime writes to the /tmp directory, so the /tmp directory must be shared between the target and sidecar container via a volume mount. This could be done, for example, by having the containers share a common [volume](https://docs.docker.com/storage/volumes/#create-and-manage-volumes) or a Kubernetes [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume. If you attempt to use the diagnostic tools from a sidecar container without sharing the /tmp directory, you will get an error about the process "not running compatible .NET runtime." +2. The tools need access to the [diagnostic port](diagnostic-port.md) Unix Domain Socket which the .NET runtime writes to the */tmp* directory, so the */tmp* directory must be shared between the target and sidecar container via a volume mount. This could be done, for example, by having the containers share a common [volume](https://docs.docker.com/storage/volumes/#create-and-manage-volumes) or a Kubernetes [emptyDir](https://kubernetes.io/docs/concepts/storage/volumes/#emptydir) volume. If you attempt to use the diagnostic tools from a sidecar container without sharing the */tmp* directory, you will get an error about the process "not running compatible .NET runtime." -If you don't want to share process namespace and the /tmp directory, many of the tools also support a more advanced --diagnostic-port command-line option. This allows you to directly specify the path to the target process' [diagnostic port](diagnostic-port.md) within the filesystem of the host or sidecar. The target container's /tmp directory will need to be mapped somewhere for this path to exist, but it doesn't have to be shared with the host or sidecar /tmp. +If you don't want to share process namespace and the */tmp* directory, many of the tools also support a more advanced `--diagnostic-port` command-line option. This allows you to directly specify the path to the target process' [diagnostic port](diagnostic-port.md) within the filesystem of the host or sidecar. The target container's /tmp directory will need to be mapped somewhere for this path to exist, but it doesn't have to be shared with the host or sidecar */tmp*. > [!NOTE] -> Even when running the diagnostic tools from the host or sidecar, the target process may still be requested to do work which increases its resource usage inside the target container. We have observed dotnet-dump may cause the target process to page in substantial virtual memory when collecting a dump. Other tools may cause smaller impacts though we haven't seen these cause problems in practice. For example dotnet-trace requests the target process to allocate an event buffer and dotnet-counters requests a small memory region where metric data is aggregated. We recommend testing to ensure your memory limits aren't so tight that running the tools causes the OS to terminate your containers. +> Even when running the diagnostic tools from the host or sidecar, the target process may still be requested to do work which increases its resource usage inside the target container. We have observed dotnet-dump may cause the target process to page in substantial virtual memory when collecting a dump. Other tools may cause smaller impacts though we haven't seen these cause problems in practice. For example, `dotnet-trace` requests the target process to allocate an event buffer and `dotnet-counters` requests a small memory region where metric data is aggregated. We recommend testing to ensure your memory limits aren't so tight that running the tools causes the OS to terminate your containers. > [!NOTE] -> When dotnet-dump writes a dump file to disk, the output path is interpretted in the context of the target process' view of the file system. This may differ from the host or sidecar container. +> When `dotnet-dump` writes a dump file to disk, the output path is interpreted in the context of the target process' view of the file system. This may differ from the host or sidecar container. ## Use `PerfCollect` in a container