README

Implementation of the paper "ZeroTracer: In-band eBPF-based Trace Generator with Zero Instrumentation for Microservice Systems".

1. Environment Setup

We build on machines running on Centos-8-based OS with Linux kernel version 5.10.134. To compile the eBPF programs, we have opted for clang and LLVM version 13.

1.1 Install clang and llvm

Install clang and llvm via command line:

$ yum install clang llvm

1.2 Install libbpf

The libbpf repository: GitHub - libbpf/libbpf. We use the source code of libbpf version 1.1.0. Move the required bpf header files used by libbpf to /usr/include/bpf/ for later compilation.

$ mkdir libbpf
$ wget https://github.com/libbpf/libbpf/archive/refs/tags/v1.1.0.tar.gz
$ tar -xzvf v1.1.0.tar.gz -C ./libbpf
$ cp -r libbpf/libbpf-1.1.0/src/ /usr/include/bpf/ # Move header files

# Build static libbpf.a and shared libbpf.so
# $ yum install elfutils-libelf-devel
$ cd libbpf/libbpf-1.1.0/src
$ make
$ cp libbpf.so*  /usr/lib/ # Move libbpf.so files to /usr/lib/

1.3 Install bazel

Refer to the official guide: https://bazel.build/install/redhat?hl=en, section "Install on CentOS 7". Here we use bazel version 4.2.1. Run the following commands:

$ vim /etc/yum.repos.d/bazel.repo
# Paste the following content
'''
[copr:copr.fedorainfracloud.org:vbatts:bazel]
name=Copr repo for bazel owned by vbatts
baseurl=https://download.copr.fedorainfracloud.org/results/vbatts/bazel/epel-7-$basearch/
type=rpm-md
skip_if_unavailable=True
gpgcheck=1
gpgkey=https://download.copr.fedorainfracloud.org/results/vbatts/bazel/pubkey.gpg
repo_gpgcheck=0
enabled=1
enabled_metadata=1
'''
$ yum install bazel4 # Install bazel 4

1.4 Install bpftool

Install bpftool with the same linux kernel version

2. Build and Run

Go to the src/ directory and run bazel build to compile the code.

$ git clone https://github.com/IntelligentDDS/zerotracer.git
$ cd zerotracer/
$ bazel build //src:tracing

To run the program, you need to specify a yaml configuration file.

cd zerotracer/
./bazel-bin/src/tracing -c [config_path]

3. End-to-End Tracing Example

Take an Go async demo (A->B->C) as an example for end-to-end tracing:

// Deployment of A->B->C services
A(Frontend)     172.16.56.157       port: 9090
B(Middle)       172.16.56.158       port: 9091
C(Backend)      172.16.56.159       port: 9092

Write config.yaml

• Add the service addresses to monitor

To trace network sockets, add the IP and port of the services to monitor. Each host's config should record the service addresses (ip:port) it listens on and the addresses it will access.

MonitorAddress:
  - ip: 172.16.56.157
    port: 9090
    name: frontend
  - ip: 172.16.56.158
    port: 9091
    name: middle

For each service address, specify:

1. ip: The IP the service listens on
2. port: The port the service listens on
3. name: The service name (used for span naming)

• Specify the target applications (Golang)

TargetExec:
  golang:
    hertz:
      - path: $PATH/hertz-demo/server/main
        engine_serve_func_offset: 0x5d5420
        runtime_newproc1_func_offset: 0x454e0
        runtime_execute_func_offset: 0x41b60
      - path: $PATH/hertz-demo/main
        engine_serve_func_offset: 0x5d5420
        runtime_newproc1_func_offset: 0x454e0
        runtime_execute_func_offset: 0x41b60
    nethttp:
      - path: $PATH/nethttp-demo/frontend/main
        c_serve_func_offset: 0x22cdc0
        runtime_newproc1_func_offset: 0x445c0
        runtime_execute_func_offset: 0x40c40
        readloop_func_offset: 0x2455c0
        writeloop_func_offset: 0x247360
        pconn_rt_func_offset: 0x2479a0

For Golang, you need to manually add configuration to mount uprobe programs to the function offsets of the target binaries.

1. For hertz-based Go programs, specify:
   • Executable path path
   • Offset of (*Engine).serve: engine_serve_func_offset
   • Offset of runtime.newproc1: runtime_newproc1_func_offset
   • Offset of runtime.execute: runtime_execute_func_offset
2. For net/http-based Go programs, specify:
   • Executable path path
   • Offset of net/http.HandlerFunc.ServeHTTP: c_serve_func_offset
   • Offset of net/http.(*persistConn).writeLoop: writeloop_func_offset
   • Offset of net/http.(*persistConn).readLoop: readloop_func_offset
   • Offset of net/http.(*persistConn).roundTrip: pconn_rt_func_offset
   • Offset of runtime.newproc1: runtime_newproc1_func_offset
   • Offset of runtime.execute: runtime_execute_func_offset

Function addresses can be obtained using objdump -t <binary> to get the symbol table. If the address in objdump is 0x674898, subtract the virtual address 0x400000 to get the function offset.