-
Notifications
You must be signed in to change notification settings - Fork 618
Debugging Utilities
Debugging multi-process applications like PostgreSQL has historically been very painful with GDB. Thankfully with recent 7.x releases, this has been improved greatly by "inferiors" (GDB's term for multiple debugged processes).
NB! This is still quite fragile, so don't expect to be able to do this in production.
## Stop server
pg_ctl -D /path/to/data stop -m fast
## Launch postgres via gdb
gdb --args peloton -D /path/to/data
Now, in the GDB shell, use these commands to set up an environment:
## We have scroll bars in the year 2015!
set pagination off
## Attach to both parent and child on fork
set detach-on-fork off
## Stop/resume all processes
set schedule-multiple on
## Usually don't care about these signals
handle SIGUSR1 noprint nostop
handle SIGUSR2 noprint nostop
## Ugly hack so we don't break on process exit
python gdb.events.exited.connect(lambda x: [gdb.execute('inferior 1'), gdb.post_event(lambda: gdb.execute('continue'))])
## Phew! Run it
run
## To get a list of processes
run info inferior
## To switch to another process
run inferior NUM
Valgrind is useful to find memory problems in Peloton.
## Try to use with something like the attached.
valgrind --suppress
valgrind \
--suppressions=$PG_SOURCE/src/tools/valgrind.supp \
--trace-children=yes --track-origins=yes --read-var-info=yes \
peloton -D REST_OF_ARGS
## Callgrind
valgrind --tool=callgrind --trace-children=yes ./src/peloton -D data &> /dev/null &
The Linux kernel comes with a performance analysis tool called perf that can even be used on production servers to analyze the call stack of a server, or a process. This is useful to find the bottlenecks of a query processing or simply of a server running and improve those.
It is usually found in the package "perf", at least it is the case of RHEL, CentOS and ArchLinux.
/proc/sys/kernel/perf_event_paranoid can be used to restrict access to the performance counters.
2, allow only user-space measurements
1, allow both kernel and user measurements (default)
0, allow access to CPU-specific data
-1, no paranoid at all
Record information
## Profile system as long as needed, can be cancelled with Ctrl-C
perf record -a -g
## Profile system for a given period of time
perf record -a -g -s sleep $TIME_IN_SECONDS
## For the duration of a command
perf record -a -g -s -- $COMMAND
Different things can be recorded:
-a to profile the whole system
-p $PID to profile only the given PID
-u $USER to profile only the given user
Reports are saved by default in $HOME/perf.data. Old reports are renamed as $HOME/perf.data.old.
## Real-time measurement is done by perf top, like system-wide profiling
perf top
## Profiling without accumulating stats
perf top -z
Viewing the records
## View profile recorded
perf report -n
## With a graph
perf report -g
## Run peloton under strace
strace -o trace.server -f -t peloton -D /path/to/data
## Run psql under strace
strace -o trace.client -f -t psql postgres
The first flag is -f, which tells strace to follow the forked processes. The -o flag sends all the output to a file. We can also use the -ff flag, that sends the output corresponding to each forked process to a separate file. Very handy, that. Finally, we add a -t flag, which prepends each line with a timestamp.
## List the peloton processes
pgrep "peloton" | xargs ps fp
## Kill all the peloton processes
pkill -9 peloton
## List all the files ending with .cpp extension
find . -name "*.cpp" -exec ls {} \;
## Replace all foo with bar
find . -name "*.cpp" -exec sed -i 's/foo/bar/' {} \;
Also, take a look at the shell script run.sh in /scripts/oltpbenchmark
- [Starting Postgres under GDB] (https://wiki.postgresql.org/wiki/Getting_a_stack_trace_of_a_running_PostgreSQL_backend_on_Linux/BSD#Starting_Postgres_under_GDB)
- [Tips and tricks from an open-source developer] (http://michael.otacoo.com/manuals/postgresql/)
- [Compilation of useful links] (https://github.com/ty4z2008/Qix/blob/master/pg.md)