Remote debug ACAP application: Part 2 - application with debug symbols

[pataxis]

Important

This tutorial is replaced by remote-debug-example for AXIS OS versions 11.11 and later.

Introduction

This is a tutorial in three parts showing how-to remote debug an ACAP application on an Axis device from a user machine.

• Part 1 - build a gdbserver
• Part 2 - application with debug symbols
• Part 3
  • Part 3a - debug the application
  • Part 3b - debug using Visual Studio Code

The setup

• Axis device (target, remote)
  • Will run gdbserver and either attach to or start an installed ACAP application to debug
• User machine (desktop)
  • Will run gdb-multiarch and connect to the gdbserver on the device.
  • Needs access to the application binary with debug symbols (not stripped)

Limitations

• This tutorial requires root access on an Axis device and this solution will only work up until next AXIS OS LTS 2024. More information in this announcement.

Part 2 - application with debug symbols

In order to debug a program effectively, you need to generate debugging information when you compile it. GDB - Compiling for Debugging.

This guide shows how to build an ACAP application container image that:

• Builds an application binary in two versions:
  • Debug binary (unstripped)
    • Contains debug symbols
    • Not part of ACAP application (.eap)
  • Application binary (stripped)
    • Doesn't contain debug symbols
    • Packaged in to the ACAP application (.eap)
• Installs program gdb-multiarch to run debug from user machine

To get a good debug experience it's often recommended to compile with -g3 -O0 to add debug information and minimize optimization. This example will use these options in the Makefile.

The application files

This example builds on the hello-world example but is here renamed to hello-debug and has some smaller modifications.

1. Clone the acap-native-sdk-example repository.
2. Copy the hello-world directory

   cp -r hello-world hello-debug
   cd hello-debug

3. Replace the files in the sections below
4. The project structure should now look like this

   hello-debug
   ├── app
   │   ├── hello_debug.c
   │   ├── LICENSE
   │   ├── Makefile
   │   └── manifest.json
   └── Dockerfile

manifest.json

{
    "schemaVersion": "1.3.1",
    "acapPackageConf": {
        "setup": {
            "appName": "hello_debug",
            "vendor": "Axis Communications",
            "embeddedSdkVersion": "3.0",
            "runMode": "never",
            "version": "1.0.0"
        }
    }
}

Makefile

Note

• This is where the debug options -g3 -O0 are added. The order is important and if multiple -g or -O options are added, the last ones will take effect. It's therefore important that these options are added after CFLAGS in the compile command.
• It's also where the debug binary (unstripped) and the application binary (stripped) are created

PROG1 = hello_debug                                                                                                                                                                                                                                                            
OBJS1 = $(PROG1).c                                                                                                                                                                                                                                                             
DEBUG_DIR = debug                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                 
PROGS = $(PROG1)                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                 
CFLAGS += -W -Wformat=2 -Wpointer-arith -Wbad-function-cast \                                                                                                                                                                                                                  
          -Winline -Wdisabled-optimization -Wfloat-equal -Wall -Werror                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                 
CFLAGS_DEBUG = -g3 -O0                                                                                                                                                                                                                                                         
                                                                                                                                                                                                                                                                                 
all:  $(PROGS)                                                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                                                                 
$(PROG1): $(OBJS1)                                                                                                                                                                                                                                                             
  # Create debug directory and build binary with debug symbols                                                                                                                                                                                                                 
  install -d $(DEBUG_DIR)
  # Make sure that CFLAGS_DEBUG comes after CFLAGS
  $(CC) $^ $(CFLAGS) $(CFLAGS_DEBUG) -o $(DEBUG_DIR)/$@                                                                                                                                                                                                                        
  # Copy out binary to application directory and strip from debug symbols                                                                                                                                                                                                      
  cp $(DEBUG_DIR)/$@ .                                                                                                                                                                                                                                                         
  $(STRIP) $@ 
                                                                                                                                                                                                                                                                                 
clean:                                                                                                                                                                                                                                                                         
  rm -rf $(PROGS) *.o core *.eap* *_LICENSE.txt debug tmp* package.conf* param.conf

Dockerfile

Note

• gdb-multiarch could also be run from a user machine without a Docker container as long as there is access to the debug binary but in this example the build and debug environment are kept together.
• The multi-stage setup here is to allow for a better development flow.

ARG ARCH=armv7hf
ARG VERSION=1.9
ARG UBUNTU_VERSION=22.04
ARG REPO=axisecp
ARG SDK=acap-native-sdk                                                                                                                                                                                                                 
 
FROM ${REPO}/${SDK}:${VERSION}-${ARCH}-ubuntu${UBUNTU_VERSION} as sdk-dev-base
 
# Install gdb-multiarch to run debug on host by connecting to gdbserver on device
RUN DEBIAN_FRONTEND=noninteractive \                                                                                                                                                                                              
    apt-get update && apt-get install -y -f --no-install-recommends \
    gdb-multiarch \
    && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /opt/app
FROM sdk-dev-base as build-app

# Build the ACAP application
COPY ./app /opt/app/
RUN . /opt/axis/acapsdk/environment-setup* && acap-build ./

hello_debug.c

Note

• A small example where a runtime error has been added, i.e. it's not seen at build time
• This file should replace hello_world.c and need to have the name hello_debug.c to build

// Copyright (C) 2023, Axis Communications AB, Lund, Sweden. All Rights Reserved.
#include <syslog.h>

void hello_fault(char* str) {
  // This will try to read outside of memory
  *(str + 1) = 'd';
  syslog(LOG_INFO, "Print fault: %s", str);
}

int a_function(char* str) {
  hello_fault(str);
  syslog(LOG_INFO, "After fault, should come nothing!");
  return 0;
}

int main() {
  char* my_char = "teste";
  openlog("hello_debug", LOG_PID | LOG_CONS, LOG_USER);
  a_function(my_char);
  return 0;
}

Build the application container image

This application can be built in the same way as the hello-world example, see the link for more detailed build steps.

docker buildx build . -t hello-debug:1.9-armv7hf
docker buildx build . --build-arg ARCH=aarch64 -t hello-debug:1.9-aarch64

Check binaries of application container image

If you're interested you can check if the different binaries have debug symbols or not.

docker run --rm -it hello-debug:1.9-armv7hf
root@container:/opt/app# file hello_debug
hello_debug: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=1234567890abcdf1234567890abcdf123456789, for GNU/Linux 3.2.0, stripped
root@container:/opt/app#
root@container:/opt/app# file debug/hello_debug
debug/hello_debug: ELF 32-bit LSB pie executable, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-armhf.so.3, BuildID[sha1]=1234567890abcdf1234567890abcdf123456789, for GNU/Linux 3.2.0, with debug_info, not stripped

Note that the binaries have the same BuildID, but one states stripped and the other one with debug_info, not stripped.

Improved development flow

This is a simple example, but for applications that build external libraries like using-opencv a better development flow can be achieved by building the libraries in a base image and handle the copying of built libraries to application project (here /opt/app) to the Makefile. This way allows running the container interactively and use command-line build command acap-build to rebuild the application during development.

docker buildx build . --target=sdk-dev-base -t hello-debug:1.9-armv7hf-dev
docker buildx build . --target=sdk-dev-base --build-arg ARCH=aarch64 -t hello-debug:1.9-aarch64-dev

Next step

Continue to debugging the application, depending on your IDE you can choose:

• Part 3a - debug the application
• Part 3b - debug using Visual Studio Code