Skip to content

Commit b6fad60

Browse files
matea16matea16
authored
update debugging page (#1341)
1 parent bccc63e commit b6fad60

File tree

1 file changed

+154
-84
lines changed

1 file changed

+154
-84
lines changed

‎pages/database-management/debugging.mdx

Lines changed: 154 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -1,79 +1,90 @@
11
---
2-
title: Debugging Memgraph by yourself
2+
title: Debugging Memgraph
33
description: Utilize tools provided to you in the container to inspect what's happening in your Memgraph instance. Send us diagnostics, so we're able to identify issues quicker and make the product more stable.
44
---
55
import { Steps } from 'nextra/components'
66
import { Callout } from 'nextra/components'
77

8-
# Debugging Memgraph by yourself
8+
# Debugging Memgraph
99

10-
### Why do we expose users with the capability of debugging Memgraph?
10+
## Why enable user debugging?
1111

12-
In the process of enhancing Memgraph's performance and reliability, user-driven debugging plays a crucial role. Given the complex nature of software environments, it is often challenging for our team to address all issues simultaneously. Moreover, bugs can exhibit behaviors that are difficult to replicate on different systems.
12+
User-driven debugging helps improve Memgraph's performance and reliability by
13+
providing diagnostic data from your environment. This data assists us in
14+
reproducing and resolving issues faster, especially for bugs that are hard to
15+
replicate.
1316

14-
By conducting initial debugging, users can provide us with valuable diagnostic data from their specific environment, which is essential for reproducing and pinpointing the root cause of issues. To facilitate this, we have equipped our container with a suite of user-friendly tools designed to assist in the debugging process.
17+
To help with this, our containers come equipped with user-friendly debugging
18+
tools, empowering you to identify and report problems more effectively.
1519

16-
These tools not only empower users to identify and report problems more accurately but also expedite the resolution process, enhancing overall system stability and performance.
20+
## Chose the right debug image
1721

18-
### Which Memgraph image can I use to debug Memgraph?
22+
Memgraph provides Docker images built in `RelWithDebInfo` mode, including tools
23+
like `perf`, `gdb` and `pgrep`. This image is about 10% slower but enables
24+
detailed debugging.
1925

20-
Memgraph offers Docker images specifically tailored for debugging, with Memgraph built in the **RelWithDebInfo** mode. These containers are equipped with crucial debugging tools such as **perf, gdb, and pgrep**, along with other useful apt packages.
26+
To pull a debug image:
2127

22-
This setup is intended to empower users to conduct thorough investigations into issues such as slow performance, system hangs, or unusually high memory usage. Although the RelWithDebInfo build mode of Memgraph comes with some performance degradations (**~10% slower performance overall**), this approach contributes to more stable and efficient system performance in the long run.
23-
24-
After identification of all the issues and moving to production, users are encouraged to switch to Release mode, with the default Memgraph images, to make sure they gain the extra performance, from the now fast and reliable system.
25-
26-
Memgraph in the RelWithDebInfo mode can be downloaded from DockerHub with the following command:
27-
28-
```bash
28+
```
2929
docker image pull <memgraph_version>-relwithdebinfo
3030
```
31-
32-
If you are using Memgraph MAGE, the following command should do
31+
For Memgraph MAGE:
3332

3433
```bash
3534
docker image pull <MAGE_version>-memgraph-<memgraph_version>-relwithdebinfo
3635
```
3736

38-
Where `memgraph_version` and `MAGE_version` are the respective versions of the image. All the images in the `RelWithDebInfo` build
39-
mode have the suffix `-relwithdebinfo`.
37+
All the images in the `RelWithDebInfo` build mode have the suffix
38+
`-relwithdebinfo`.
39+
40+
## Run Memgraph in debug mode
4041

41-
### I have the Memgraph image, is there anything specific I should do to run Memgraph?
42-
Yes, Memgraph container needs to be run in the **--privileged mode**. Privileged mode is an option of the Docker ecosystem, not Memgraph. It will allow for gdb and perf tools to run properly.
42+
Run the Memgraph container in privileged mode to allow debugging tools like
43+
`gdb` and `perf` to function:
4344

44-
Below, we can see an example command of how to run a Memgraph container in the privileged mode:
45+
Below, we can see an example command of how to run a Memgraph container in the
46+
privileged mode:
4547

4648
```bash
4749
docker container run --name mg --privileged -p 7687:7687 -p 9091:9091 memgraph --log-level=TRACE --also-log-to-stderr
4850
```
4951

50-
### I have run Memgraph container, where should I perform the debugging?
51-
All debugging is performed inside the container. To enter the container, you need to execute the
52-
following command.
52+
## Accessing the container
53+
54+
All debugging is performed inside the container. To enter the container, you
55+
need to execute the following command.
56+
5357
```bash
5458
docker container exec -it -u root memgraph bash
5559
```
5660

57-
The ```-u root``` command is there to enable root privileges inside the container, necessary for running debugging tools.
61+
The ```-u root``` command is there to enable root privileges inside the
62+
container, necessary for running debugging tools.
63+
64+
65+
## Debugging tools overview
5866

59-
### What are all the debugging capabilities I am equipped with?
6067
Memgraph supports the following debug capabilities:
61-
1. Attaching Memgraph with `GDB` and inspecting threads
62-
2. Generating a core dump after Memgraph crashed
63-
3. Using `perf` to identify performance bottlenecks
68+
1. [Using GDB](#using-gdb): Attaching Memgraph with `GDB` and inspecting threads
69+
2. [Generating a core dump](#generating-core-dump-via-docker) after Memgraph crashed
70+
3. [Using `perf`](#profiling-with-perf) to identify performance bottlenecks
71+
6472

65-
we will be going through each of them in detail.
73+
### Using GDB
6674

67-
## 1. Attaching Memgraph with GDB
68-
`GDB` and `pgrep` are already installed packages in the Memgraph container that has the debug symbols.
69-
Since Memgraph is already running there at port 7687, you can attach GDB to your running Memgraph with the following command:
75+
`GDB` and `pgrep` are already installed packages in the Memgraph container that
76+
has the debug symbols. Since Memgraph is already running there at port `7687`,
77+
you can attach `GDB` to your running Memgraph with the following command:
7078

7179
```bash
7280
gdb -p $(pgrep memgraph)
7381
```
74-
Most likely, the Memgraph process will have the PID number 1, but for certainty, we use `pgrep`.
7582

76-
### List of useful commands when in GDB
83+
Most likely, the Memgraph process will have the PID number 1, but for certainty,
84+
we use `pgrep`.
85+
86+
**Useful GDB commands:**
87+
7788

7889
| Name | Description |
7990
| ----------------- | ------------------------------------------------------------------- |
@@ -91,14 +102,17 @@ Most likely, the Memgraph process will have the PID number 1, but for certainty,
91102
| `info args` | Prints function arguments of the current frame. |
92103
| `print $local_var`| Prints the value of the variable `$local_var`. |
93104

94-
### Most commonly used commands.
95-
In Memgraph, we usually want to see first what are all the threads currently in place. We do that by issuing:
105+
106+
107+
In Memgraph, we usually want to see first what are all the threads currently in
108+
place. We do that by issuing:
96109

97110
```bash
98111
info thread
99112
```
100113

101-
By identifying a certain thread with a code that could belong to the Memgraph repository, we can issue the command
114+
By identifying a certain thread with a code that could belong to the Memgraph
115+
repository, we can issue the command
102116

103117
```bash
104118
t <x>
@@ -110,26 +124,36 @@ Seeing the backtrace can be done with the command
110124
bt
111125
```
112126

113-
## 2. Generating a core dump with Memgraph
127+
### Generating core dump via Docker
114128

115-
### Generating a core dump with a plain Docker image
116-
In order to generate a core dump, you need to do a few steps on the host and in the container image.
129+
In order to generate a core dump, you need to do a few steps on the host and in
130+
the container image.
117131

118-
1. Setting no size limit to the core dump
119-
Initially, you will need to set no boundaries to the size of the core dump that can be generated. This is done with the following command:
132+
<Steps>
133+
{<h4 className="custom-header">Set no size limit to the core dump</h4>}
134+
135+
Initially, you will need to set no boundaries to the size of the core dump that
136+
can be generated. This is done with the following command:
120137

121138
```bash
122139
ulimit -c unlimited
123140
```
124141

125-
2. Mounting the correct volume
126-
When Memgraph crashes, we would want to get the present core dump file on our host system. When starting the container, we will provide the appropriate volume. Additionally, don't forget to set the `--privileged` flag as noted in the previous sections.
142+
{<h4 className="custom-header">Mount the correct volume</h4>}
143+
144+
When Memgraph crashes, we would want to get the present core dump file on our
145+
host system. When starting the container, we will provide the appropriate
146+
volume. Additionally, don't forget to set the `--privileged` flag as noted in
147+
the previous sections.
148+
127149
```bash
128150
docker container run --name mg --privileged -v /home/user/cores:/tmp/cores -p 7687:7687 -p 9091:9091 memgraph:2.16.0_17_050d5c985 --log-level=TRACE --also-log-to-stderr
129151
```
130152

131-
3. Setting up the container for core dump generation
132-
Additionally, in the container, the following commands will need to be executed after the container has started, to be able to generate a correct core dump.
153+
{<h4 className="custom-header">Set up the container</h4>}
154+
155+
Additionally, in the container, the following commands will need to be executed
156+
after the container has started, to be able to generate a correct core dump.
133157

134158
```bash
135159
ulimit -c unlimited
@@ -138,14 +162,21 @@ chmod a+rwx /tmp/cores
138162
echo "/tmp/cores/core.%e.%p.%h.%t" > /proc/sys/kernel/core_pattern
139163
```
140164

141-
When Memgraph crashes, a core dump will be generated, and you will see it on the host system if you have mounted the volume correctly.
165+
When Memgraph crashes, a core dump will be generated, and you will see it on the
166+
host system if you have mounted the volume correctly.
167+
168+
{<h4 className="custom-header">Inspecting the container</h4>}
142169

143-
4. Starting Memgraph again and inspecting the core dump with `GDB`
144-
The container will need to be started again since we want the same debug symbols to be present, and using an identical image is the most proper way for that. However, we don't need now the Memgraph process at port 7687, so we will ignore it.
170+
The container will need to be started again since we want the same debug symbols
171+
to be present, and using an identical image is the most proper way for that.
172+
However, we don't need now the Memgraph process at port 7687, so we will ignore
173+
it.
145174

146-
You will need to copy the core dump file into the container with the `docker cp` command.
175+
You will need to copy the core dump file into the container with the `docker cp`
176+
command.
147177

148178
After logging into the container with the root credentials:
179+
149180
```bash
150181
docker container exec -it -u root memgraph bash
151182
```
@@ -156,12 +187,20 @@ we will execute `GDB` with the core dump file provided
156187
gdb /usr/lib/memgraph/memgraph --core=/core.memgraph.file
157188
```
158189

159-
where the `core.memgraph.file` is the name of your core dump file. Possibly, appropriate permissions will need to be set on the core dump file. You can check the list of useful `GDB` commands in the above sections.
190+
where the `core.memgraph.file` is the name of your core dump file. Possibly,
191+
appropriate permissions will need to be set on the core dump file. You can check
192+
the list of useful `GDB` commands in the above sections.
160193

161-
To find out more about setting core dumps, you can check [this article](https://medium.com/@sourabhedake/core-dumps-how-to-enable-them-73856a437711).
194+
To find out more about setting core dumps, you can check [this
195+
article](https://medium.com/@sourabhedake/core-dumps-how-to-enable-them-73856a437711).
162196

163-
### Generating a core dump with Docker Compose
164-
The setup with Docker Compose is similar to Docker. You will need to bind the volume, run Memgraph in privileged mode, and make sure you set no size limit on the generated core dump.
197+
</Steps>
198+
199+
### Generating core dump via Docker Compose
200+
201+
The setup with Docker Compose is similar to Docker. You will need to bind the
202+
volume, run Memgraph in privileged mode, and make sure you set no size limit on
203+
the generated core dump.
165204

166205
Below we can see an example Docker Compose file which can generate a core dump:
167206

@@ -196,53 +235,78 @@ services:
196235
```
197236
198237
199-
## 3. Using `perf` to identify performance bottlenecks
238+
### Profiling with `perf`
200239

201-
Perfing is the most common operation that is run when Memgraph is hanging or performing slowly.
240+
Perfing is the most common operation that is run when Memgraph is hanging or
241+
performing slowly.
202242

203-
In the next steps, the instructions are provided on how to check which parts of Memgraph are stalling during query execution, so we can use the information to make the system better.
243+
In the next steps, the instructions are provided on how to check which parts of
244+
Memgraph are stalling during query execution, so we can use the information to
245+
make the system better.
204246

205-
Prior to performing perf instructions, you will need to bound the Memgraph binary to the local filesystem.
206-
You can start Memgraph with the volume binded like this:
247+
Prior to performing perf instructions, you will need to bound the Memgraph
248+
binary to the local filesystem. You can start Memgraph with the volume binded
249+
like this:
207250

208251
```bash
209252
docker container run --name mg -p 7687:7687 --privileged -v memgraph-binary:/usr/lib/memgraph <memgraph_image> --log-level=TRACE --also-log-to-stderr
210253
```
211254

212-
1. Perfing Memgraph
213-
Inside the container, we will need to perf the system. That is done using the following command.
255+
<Steps>
256+
{<h4 className="custom-header">Record performance data</h4>}
257+
258+
Perfing Memgraph Inside the container, we will need to perf the system. That is
259+
done using the following command.
214260

215261
```bash
216262
perf record -p $(pgrep memgraph) --call-graph dwarf sleep 5
217263
```
218264

219-
The command will perf the Memgraph process for 5 seconds. Of course, you can tweak the number by yourself. The command will generate a file called `perf.data` in the directory you have performed the command in the container
265+
The command will perf the Memgraph process for 5 seconds. Of course, you can
266+
tweak the number by yourself. The command will generate a file called
267+
`perf.data` in the directory you have performed the command in the container
220268

221-
2. Installing `hotspot` as the GUI tool on the host
222-
On the outside, we will need to install a GUI tool called `hotspot`, that will help us generate a [flamegraph](https://www.brendangregg.com/flamegraphs.html). We can install `hotspot` on the host machine by issuing the command:
269+
270+
{<h4 className="custom-header">Install hotspot (GUI tool)</h4>}
271+
272+
We will need to install a GUI tool called `hotspot`, that will help us generate
273+
a [flamegraph](https://www.brendangregg.com/flamegraphs.html). We can install
274+
`hotspot` on the host machine by issuing the command:
223275

224276
```bash
225277
apt install hotspot
226278
```
227279

228-
if your machine does not support APT, please check the [hotspot repo](https://github.com/KDAB/hotspot) and follow the installation steps.
280+
If your machine does not support APT, please check the [hotspot
281+
repo](https://github.com/KDAB/hotspot) and follow the installation steps.
282+
283+
284+
{<h4 className="custom-header">Copy `perf.data` to host</h4>}
285+
286+
After we have perfed the data, we need to get the perf information from the
287+
container to the host. That is done with the following command **on the host**
229288

230-
3. Transferring `perf.data` to host
231-
After we have perfed the data, we need to get the perf information from the container to the host. That is done with the following command **on the host**
232289
```bash
233290
docker cp <container_id>:<path_to_perf.data> .
234291
```
235292

236-
where the `container_id` is the Memgraph container ID, and the `path_to_perf.data` is the absolute path in the container to the generated `perf.data` file.
293+
where the `container_id` is the Memgraph container ID, and the
294+
`path_to_perf.data` is the absolute path in the container to the generated
295+
`perf.data` file.
237296

238-
4. Linking Memgraph's debug symbols in the container to the host
239-
For `hotspot` to be able to identify debug symbols and draw the flamegraph, it needs the path to the debug symbols to be the exact one as in the container. In the container it is the `/usr/lib/memgraph/memgraph` binary, so we will need to make a symbolic link from the container volume to the host system:
297+
{<h4 className="custom-header">Link debug symbols</h4>}
298+
299+
For `hotspot` to be able to identify debug symbols and draw the flamegraph, it
300+
needs the path to the debug symbols to be the exact one as in the container. In
301+
the container it is the `/usr/lib/memgraph/memgraph` binary, so we will need to
302+
make a symbolic link from the container volume to the host system:
240303

241304
```bash
242305
ln -s <path_to_docker_volume_debug_symbols_binary> /usr/lib/memgraph/memgraph
243306
```
244307

245-
5. Start `hotspot`
308+
{<h4 className="custom-header">Open flamegraph</h4>}
309+
246310
If you did everything correctly, by starting hotspot
247311

248312
```bash
@@ -253,6 +317,8 @@ you should be able to see a similar flamegraph like in the picture below.
253317

254318
![](/pages/database-management/debugging/perf.png)
255319

320+
</Steps>
321+
256322
## Debugging Memgraph under Kubernetes (k8s)
257323

258324
### General commands
@@ -412,29 +478,33 @@ To enable core dumps, create a `values.yaml` file with at least the following se
412478
createCoreDumpsClaim: true
413479
```
414480

415-
Setting this value to true will also enable the use of GDB inside Memgraph containers when using our provided [charts](https://github.com/memgraph/helm-charts).
481+
Setting this value to true will also enable the use of GDB inside Memgraph
482+
containers when using our provided
483+
[charts](https://github.com/memgraph/helm-charts).
416484

417-
<Callout type="info">
418-
Feel free to copy values file from the [helm-charts repository](https://github.com/memgraph/helm-charts) as a base, since additional required fields may be missing from a minimal config.
419-
</Callout>
485+
<Callout type="info"> Feel free to copy values file from the [helm-charts
486+
repository](https://github.com/memgraph/helm-charts) as a base, since additional
487+
required fields may be missing from a minimal config. </Callout>
420488

421-
This instructs the Helm chart to create a `PersistentVolumeClaim` (PVC) to
422-
store core dumps generated by the Memgraph process.
489+
This instructs the Helm chart to create a `PersistentVolumeClaim` (PVC) to store
490+
core dumps generated by the Memgraph process.
423491

424492
{<h4 className="custom-header">Important configuration notes</h4>}
425493

426-
**By default the storage size is 10GiB**. Core dumps can be as large as your node's total RAM, so it's recommended to set this explicitly and make sure to adjust the `coreDumpsStorageSize` under
427-
`values.yaml` file.
494+
**By default the storage size is 10GiB**. Core dumps can be as large as your
495+
node's total RAM, so it's recommended to set this explicitly and make sure to
496+
adjust the `coreDumpsStorageSize` under `values.yaml` file.
428497

429-
**Make sure to use the `relwithdebinfo` image** of Memgraph by setting the `image.tag` also under `values.yaml` file.
498+
**Make sure to use the `relwithdebinfo` image** of Memgraph by setting the
499+
`image.tag` also under `values.yaml` file.
430500

431501
Run the following command to install Memgraph with the debugging configuration:
432502
```
433503
helm install my-release memgraph/memgraph -f values.yaml
434504
```
435505

436-
The core dumps are written to a mounted volume inside the container (the
437-
default is `/var/core/memgraph`, it's possible to tweak that by changing the
506+
The core dumps are written to a mounted volume inside the container (the default
507+
is `/var/core/memgraph`, it's possible to tweak that by changing the
438508
`coreDumpsMountPath` under `values.yaml`). You can use `kubectl exec` or
439509
`kubectl cp` to access the files for post-mortem analysis.
440510

0 commit comments

Comments
 (0)