Remote debug ACAP application: Part 3a - debug the application

[pataxis]

Important

• This tutorial is replaced by remote-debug-example for AXIS OS versions 11.11 and later.
• Some aspects of this tutorial might still be of interest:
  • debugging directly with gdb-multiarch instead of using Visual Studio Code.
  • different ways to connect gdbserver to an application, both by starting the application and attaching to a running one.

Introduction

This is a tutorial in three parts showing how-to remote debug an ACAP application on an Axis device from a user machine.

• Part 1 - build a gdbserver
• Part 2 - application with debug symbols
• Part 3
  • Part 3a - debug the application
  • Part 3b - debug using Visual Studio Code

The setup

• Axis device (target, remote)
  • Will run gdbserver and either attach to or start an installed ACAP application to debug
• User machine (desktop)
  • Will run gdb-multiarch and connect to the gdbserver on the device.
  • Needs access to the application binary with debug symbols (not stripped)

Limitations

• This tutorial requires root access on an Axis device and this solution will only work up until next AXIS OS LTS 2024. More information in this announcement.

Part 3a - debug the application

This guide shows how to use the gdbserver from part 1 and the application binary with debug symbols from part 2 to debug an installed ACAP application.

Two variants will be shown that can suit different use cases:

• Debug an application that fails on startup
• Debug a running application that fails on a certain event (e.g. a CGI call)

Development environment

The development environment consists of running the container hello-debug:1.9-armv7hf-dev interactively with the application project in $PWD/app being mounted in to /opt/app.

The default user in the container is root which means that all build artifacts in the application project will be owned by root. One way to avoid this is to start the container as the user and mount in user and group information. Here's a minimal example:

cd hello_debug
docker run --rm \
  -u $(id -u):$(id -g) \
  -v /etc/passwd:/etc/passwd \
  -v /etc/group:/etc/group \
  -v "$PWD/app":/opt/app
  -it hello-debug:1.9-armv7hf-dev

This allows the following workflow:

• Terminal 1: User machine, application container - build environment
  • Inside the container in the application project /opt/app is where you can build acap-build and deploy eap-install.sh the application.
• Terminal 2: Axis device - gdbserver
  • SSH in to device and start the gdbserver.
• IDE: User machine - change application code
  • Make changes in the application code in application project $PWD/app with an IDE.

Debug an application that fails on startup

On user machine

Start the application container interactively in a first terminal. The entry point is /opt/app where the application project is found.
Build the ACAP application with acap-build ..

cd hello_debug
docker run --rm \
  -u $(id -u):$(id -g) \
  -v /etc/passwd:/etc/passwd \
  -v /etc/group:/etc/group \
  -v "$PWD/app":/opt/app
  -it hello-debug:1.9-armv7hf-dev
myuser@container:/opt/app# acap-build .

The build files contains two files of interest for this tutorial:

• The ACAP application - /opt/app/hello_debug_1_0_0_armv7hf.eap
• The debug binary - /opt/app/debug/hello_debug

Install the ACAP application but don't start it.

Note

• Password from root user is required at install.

myuser@container:/opt/app# eap-install.sh 192.168.0.90 <password> install

On Axis device

In a second terminal, SSH in to device as root. Start gdbserver with the application binary installed under /usr/local/packages/APPNAME/BINARYNAME and open a TCP connection to a debug port on the Axis device, here port 1234 is used.

ssh root@192.168.0.90
root@axisdevice$ /tmp/gdbserver :1234 /usr/local/packages/hello_debug/hello_debug
Process /usr/local/packages/hello_debug/hello_debug created; pid = 21919
Listening on port 1234

The gdbserver will stop at the entry point of the application to let you add breakpoints and start stepping through the application from the user machine.

On user machine

Return to the terminal with the application container. Start gdb-multiarch and point to the binary with debug symbols. The program takes commands and has the prompt (gdb).

myuser@container:/opt/app# gdb-multiarch -q debug/hello_debug
Reading symbols from debug/hello_debug... 
(gdb)

Connect to gdbserver running on the Axis device.

(gdb) target remote 192.168.0.90:1234
<<<OUTPUT>>>

Note in the second terminal where gdbserver is running that a connection has been established from the user machine.

Remote debugging from host ::ffff:12.10.0.120., port 34567

Now you can add breakpoints with b and step through the application with c. Thanks to the debug symbols gdb knows for example function names. Try to type b mai and then tab, which should expand to b main, finish with enter to add the breakpoint.

Test adding these breakpoints.

(gdb) b main
(gdb) b a_function
(gdb) b hello_fault

and step through the program with c until you reach the error which will stop the application. You can at any breakpoint see how the ACAP application behaves on the device and if it's behaving as expected or not.

Debug a running application that fails on a certain event (e.g. a CGI call)

Note

• The example application from part 2 will not work in this setup since it reaches an error and finishes, which means it's not possible to attach to.
• A more realistic case would be to try to debug the web-server-using-fastcgi example by adding the build of a debug binary in the Makefile (CFLAGS_DEBUG must come after CFLAGS) the same way as hello_debug was done.
• We'll assume in this example that web-server-using-fastcgi has been built in to an application container image of tag fastcgi:armv7hf with the above mentioned debug binary.

On user machine

Start the application container interactively in a first terminal. The entry point is /opt/app where the application project is found.
Build the ACAP application with acap-build ..

cd hello_debug
docker run --rm \
  -u $(id -u):$(id -g) \
  -v /etc/passwd:/etc/passwd \
  -v /etc/group:/etc/group \
  -v "$PWD/app":/opt/app
  -it hello-debug:1.9-armv7hf-dev
myuser@container:/opt/app# acap-build .

The build files contains two files of interest for this tutorial:

• The ACAP application - /opt/app/hello_debug_1_0_0_armv7hf.eap
• The debug binary - /opt/app/debug/hello_debug

Note

• Password from root user is required at install.

Install and start the ACAP application.

myuser@container:/opt/app# eap-install.sh 192.168.0.90 <password> install
myuser@container:/opt/app# eap-install.sh start

On Axis device

In a second terminal, SSH in to device as root. Start gdbserver, attach to the running application hello_debug and open a TCP connection to a debug port on the Axis device, here port 1234 is used.

ssh root@192.168.0.90
root@axisdevice$ /tmp/gdbserver --attach :1234 $(pidof fastcgi_example)
Attached; pid = 46211
Listening on port 1234

Now you can connect and add breakpoints related to the CGI code from the user machine.

On user machine

Return to the terminal with the application container. Start gdb-multiarch and point to the binary with debug symbols. The program takes commands and has the prompt (gdb).

myuser@container:/opt/app# gdb-multiarch -q debug/fastcgi_example
Reading symbols from debug/fastcgi_example... 
(gdb)

Connect to gdbserver running on the Axis device and is attached to ACAP application.

(gdb) target remote 192.168.0.90:1234
<<<OUTPUT>>>

Note in the second terminal where gdbserver is running that a connection has been established from the user machine.

Remote debugging from host ::ffff:12.10.0.120., port 56734

Now you can add breakpoints and debug, e.g. add the following breakpoints

(gdb) b uriParseSingleUriA@plt
(gdb) b FCGX_Finish_r@plt

and make a call to the CGI, e.g. by entering the web page with URL:

http://192.168.0.90/local/fastcgi_example/example.cgi?name=DebuggingFastCGI

Notice that the web page is loading until you have stepped through all the breakpoints.

Documentation

• gdbserver
• gdb