esp-idf-net-logging

Redirect esp-idf logging to the network.

esp-idf has a Logging library.
The Logging library contains the esp_log_set_vprintf function.
By default, log output goes to UART0.
This function can be used to redirect log output to some other destination, such as file or network.

I made a project that uses 3 UARTs of ESP32.
But I was in trouble because there was no logging output destination.
So I made this.

The following protocols are available for this project.

• UDP
• TCP
• MQTT
• HTTP(POST)
• SSE

We can use Linux's rsyslogd as the logger.
I referred to this.

Software requirements

ESP-IDF V5.0 or later.
ESP-IDF V4.4 release branch reached EOL in July 2024.
ESP-IDF V5.1 is required when using ESP32-C6.

Installation

git clone https://github.com/nopnop2002/esp-idf-net-logging
cd esp-idf-net-logging/basic
idf.py menuconfig
idf.py flash

Configuration

Configuration for UDP Redirect

ESP32 works as a UDP client.

There are the following four methods for specifying the UDP Address.

• Limited broadcast address
  The address represented by 255.255.255.255, or <broadcast>, cannot cross the router.
  Both the sender and receiver must specify a Limited broadcast address.

• Directed broadcast address
  It is possible to cross the router with an address that represents only the last octet as 255, such as 192.168.10.255.
  Both the sender and receiver must specify the Directed broadcast address.
  Note that it is possible to pass through the router.

• Multicast address
  Data is sent to all PCs belonging to a specific group using a special address (224.0.0.0 to 239.255.255.255) called a multicast address.
  I've never used it, so I don't know anything more.

• Unicast address
  It is possible to cross the router with an address that specifies all octets, such as 192.168.10.41.
  Both the sender and receiver must specify the Unicast address.

Configuration for TCP Redirect

ESP32 works as a TCP client.
You can use the mDNS hostname (tcp-server.local) instead of the IP address.
If esp32 can't connect to a TCP server, it won't redirect.

Configuration for MQTT Redirect

ESP32 works as a MQTT client.
If esp32 can't connect to a MQTT broker, it won't redirect.

Configuration for HTTP Redirect

ESP32 works as a HTTP client.
You can use an mDNS hostname instead of an IP address.
If esp32 can't connect to a HTTP server, it won't redirect.

Configuration for SSE Redirect

ESP32 works as a SSE server.

Disable Logging to STDOUT

Use xRingBuffer as IPC

Both xMessageBuffer and xRingBuffer are interprocess communication (IPC) components provided by ESP-IDF.
Several drivers provided by ESP-IDF use xRingBuffer.
This project uses xMessageBuffer by default.
If you use this project at the same time as a driver that uses xRingBuffer, using xRingBuffer uses less memory.
Memory usage status can be checked with idf.py size-files.

View logging

You can view the logging using python code or various tools.

• for UDP
  
  You can use netcat as server.
  
  We can use this as Logging Viewer.
  Note that the most recent logging is displayed at the top.
  
  We can also use this as Logging Viewer.
  Note that the most recent logging is displayed at the buttom.
  I like this one better.
  
  There are others if you look for them.

• for TCP
  
  You can use netcat as server.
  
  We can use this as Logging Viewer.
  

• for MQTT
  The wifi logging is output in two parts.
  First time:W (7060) wifi:
  Second time:Characters after that
  In MQTT and HTTP, it is displayed separately in two.
  If you use broker.emqx.io, continuous Logging will drop.
  
  Using a local MQTT server is stable.
  You can use this as a broker.
  

• for HTTP
  

• for SSE
  Open a browser and enter the IP address of the ESP32 in the address bar.

Using linux rsyslogd as logger

We can forward logging to rsyslogd on Linux machine.
Configure with protocol = UDP and port number = 514.

The rsyslog server on linux can receive logs from outside.
Execute the following command on the Linux machine that will receive the logging data.
Please note that port 22 will be closed when you enable ufw.
I used Ubuntu 22.04.

$ cd /etc/rsyslog.d/

$ sudo vi 99-remote.conf
module(load="imudp")
input(type="imudp" port="514")

if $fromhost-ip != '127.0.0.1' and $fromhost-ip != 'localhost' then {
    action(type="omfile" file="/var/log/remote")
    stop
}

$ sudo ufw enable
Firewall is active and enabled on system startup

$ sudo ufw allow 514/udp
Rule added
Rule added (v6)

$ sudo ufw allow 22/tcp
Rule added
Rule added (v6)

$ sudo systemctl restart rsyslog

$ ss -nulp | grep 514
UNCONN 0      0            0.0.0.0:514        0.0.0.0:*
UNCONN 0      0               [::]:514           [::]:*

$ sudo ufw status
Status: active

To                         Action      From
--                         ------      ----
514/udp                    ALLOW       Anywhere
22/tcp                     ALLOW       Anywhere
514/udp (v6)               ALLOW       Anywhere (v6)
22/tcp (v6)                ALLOW       Anywhere (v6)

Logging from esp-idf goes to /var/log/remote.

$ tail -f /var/log/remote
May  8 14:06:09 I (6688) MAIN: This is info level
May  8 14:06:09 W (6698) MAIN: This is warning level
May  8 14:06:09 E (6698) MAIN: This is error level
May  8 14:06:09 I (6698) MAIN: freeRTOS version:V10.5.1
May  8 14:06:09 I (6708) MAIN: NEWLIB version:4.3.0
May  8 14:06:09 I (6708) MAIN: lwIP version:2-2-0-0
May  8 14:06:09 I (6708) MAIN: ESP-IDF version:v5.4.1-dirty
May  8 14:06:09 I (6718) MAIN: chip model is 1,
May  8 14:06:09 I (6718) MAIN: chip with 2 CPU cores, WiFi/BT/BLE
May  8 14:06:09 I (6718) MAIN: silicon revision 100
May  8 14:06:09 I (6728) MAIN: 2MB external flash
May  8 14:06:09 I (6728) main_task: Returned from app_main()
May  8 14:06:09 I (7398) wifi:
May  8 14:06:09 192.168.10.130  <ba-add>idx:0 (ifx:0, f8:b7:97:36:de:52), tid:5, ssn:0, winSize:64
May  8 14:06:09 192.168.10.130
May  8 14:06:10 I (7498) wifi:
May  8 14:06:10 192.168.10.130  <ba-add>idx:1 (ifx:0, f8:b7:97:36:de:52), tid:0, ssn:4, winSize:64
May  8 14:06:10 192.168.10.130

One advantage of using rsyslogd is that you can take advantage of log file rotation.
Rotating log files prevents the log files from growing forever.
The easiest way to rotate logs is to add /var/log/remote to /etc/logrotate.d/rsyslog.
There are many resources available on the Internet about rotating log files.

$ ls -l /var/log/remote*
-rw-r--r-- 1 syslog syslog    6264 Jun  6 10:35 /var/log/remote
-rw-r--r-- 1 syslog syslog 1219823 Jun  6 10:33 /var/log/remote.1

Disable ANSI Color control

You can disable this if you are unable to display ANSI color codes correctly.

Enable ANSI Color control

Disable ANSI Color control

API

Use one of the following.
Subsequent logging will be redirected.

esp_err_t udp_logging_init(char *ipaddr, unsigned long port, int16_t enableStdout);
esp_err_t tcp_logging_init(char *ipaddr, unsigned long port, int16_t enableStdout);
esp_err_t mqtt_logging_init(char *url, char *topic, int16_t enableStdout);
esp_err_t http_logging_init(char *url, int16_t enableStdout);
esp_err_t sse_logging_init(unsigned long port, int16_t enableStdout);

It is possible to use multiple protocols simultaneously.
The following example uses UDP and SSE together.

udp_logging_init("255.255.255.255", 6789, true);
sse_logging_init(8080, true);

How to use this component in your project

Create idf_component.yml in the same directory as main.c.

YourProject --+-- CMakeLists.txt
              +-- main --+-- main.c
                         +-- CMakeLists.txt
                         +-- idf_component.yml

Contents of idf_component.yml.

dependencies:
  nopnop2002/net_logging:
    path: components/net-logging/
    git: https://github.com/nopnop2002/esp-idf-net-logging.git

When you build a projects esp-idf will automaticly fetch repository to managed_components dir and link with your code.

YourProject --+-- CMakeLists.txt
              +-- main --+-- main.c
              |          +-- CMakeLists.txt
              |          +-- idf_component.yml
              +-- managed_components ----- nopnop2002__net_logging