ESP32-S3 SPIRAM Address Space

[EricNRS]

I am testing SPIRAM on the ESP32-S3 and while trying to track down the cause of memory exceptions when using external SPIRAM, I have gotten a bit confused in regards to the SPIRAM address range.

The DT shows the SPIRAM starting at 0x3c000000 and growing up (which matches the TRM for external memory).

zephyr/dts/xtensa/espressif/esp32s3/esp32s3_wroom_n16r8.dtsi

Lines 14 to 18 in 32bd3aa

	/* 8MB psram */
	&psram0 {
	reg = <0x3c000000 DT_SIZE_M(8)>;
	status = "okay";
	};

However, the linker script is showing that the SPIRAM ends at 0x3E000000 and starts at 0x3E000000 minus the length. This does match a quick read/write test, so I suspect that the DT is wrong in this case (or only correct when the SPIRAM is 32MB).

zephyr/soc/xtensa/espressif_esp32/esp32s3/default.ld

Line 42 in 32bd3aa

#define EXT_RAM_ORG (0x3E000000 - CONFIG_ESP_SPIRAM_SIZE)

zephyr/soc/xtensa/espressif_esp32/esp32s3/default.ld

Lines 79 to 81 in 32bd3aa

	#if defined(CONFIG_ESP_SPIRAM)
	ext_ram_seg(RWX): org = EXT_RAM_ORG, len = CONFIG_ESP_SPIRAM_SIZE
	#endif

Also, how can I check the MMU configuration? I haven't been able to track that down yet and could use some pointers.

Adding @sylvioalves and @LucasTambor since they have done some support/changes related to this.

[marekmatej]

@EricNRS, The correct starting address of the external data memory for the S3 is 0x3c000000, check the ESP32S3 TRM chapter 4.3.3.1. The linker script implementation is mapping the PSram from its upper limit. The reason for that is the Dcache mapping rodata segment from a lower limit of that address space.

[EricNRS]

Thanks Marek, that clarifies why the mapping PSRAM mapping is done towards the end.

The issue I am trying to debug is a networking crash when sending large packets. I did some test code to to allocate from the SPIRAM
void *ptr = k_malloc(2 * CONFIG_ESP_HEAP_MIN_EXTRAM_THRESHOLD);

And the failure is in heap.h. A10 looks like the correct beginning of the SPIRAM (0x3E000000 - 8MB), so not sure why it is crashing.

zephyr/lib/os/heap.h

Line 114 in 32bd3aa

return ((uint32_t *)cmem)[f];

[00:25:39.931,000] <err> os:  **  PC 0x421e6e58 VADDR 0x92d55554
[00:25:39.931,000] <err> os:  **  PS 0x60820
[00:25:39.931,000] <err> os:  **    (INTLEVEL:0 EXCM: 0 UM:1 RING:0 WOE:1 OWB:8 CALLINC:2)
[00:25:39.931,000] <err> os:  **  A0 0x821e6f98  SP 0x3fca28f0  A2 0xaaaaaaaa  A3 0x802
[00:25:39.931,000] <err> os:  **  A4 (nil)  A5 (nil)  A6 0x3fca2ad8  A7 (nil)
[00:25:39.931,000] <err> os:  **  A8 0x14  A9 0x7fff A10 0x3d800000 A11 0x7ffff
[00:25:39.931,000] <err> os:  ** A12 0xb A13 0x3 A14 0xaaaaaaaa A15 0x92d55550
[00:25:39.931,000] <err> os:  ** LBEG (nil) LEND (nil) LCOUNT (nil)
[00:25:39.931,000] <err> os:  ** SAR 0x1b

The coredump doesn't have the register values, just the heap.h line 114 in the backtrace.

SPIRAM itself seems to be fine as I can read and write values from 0x3d800000 to 0x3e000000 which is the correct 8 MB value.

Unfortunately, I haven't been able to debug this successfully yet as GDB ends up switching to core 1 every time I step through the code. I wonder if I can disable core 1 for this test as I don't need WiFi.

Any tips on how best to debug this? The tests samples/boards/esp32/spiram_test and test/boards/espressif_esp32/cache_coex both pass, but allocating from the external heap immediate crashes.

Here is the heap configuration from zephyr/build/.config:

CONFIG_SPIRAM=y
CONFIG_ESP_SPIRAM_SIZE=8388608

CONFIG_HEAP_MEM_POOL_SIZE=56320
CONFIG_ESP_HEAP_MIN_EXTRAM_THRESHOLD=8192
CONFIG_ESP_HEAP_SEARCH_ALL_REGIONS=y
CONFIG_ESP_HEAP_MEM_POOL_REGION_1_SIZE=49152

CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM=y

[EricNRS]

I put k_sched_lock() / unlock() around the k_malloc and debugging is going better and I should have some more details on the failure case either tonight or tomorrow.

[EricNRS]

I'm seeing what looked like memory corruption that was causing the crash. However, checking the map file, the issue is caused by the SPIRAM heap and the WiFi heaps being allocated on top of each other when CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM is enabled which results in the system heap linked-list getting corrupted. Disabling this option allowed heap allocations of k_malloc(2 * CONFIG_ESP_HEAP_MIN_EXTRAM_THRESHOLD) to be successful and the pointers were in the expected 0x3d800000 to 0x3E000000 range.

Here is part of the map file showing the overlap of the system heap (heap_caps.c):

 .ext_ram.bss.1
                0x000000003d800000        0x4 zephyr/libzephyr.a(heap_caps.c.obj)
                0x000000003d800000                _spiram_data_start
                0x000000003d800004                . = ALIGN (0x4)
                0x000000003d800004                _ext_ram_bss_end = ABSOLUTE (.)

.ext_ram_noinit
                0x000000003d800010    0x4c5e4
 *libdrivers__wifi.a:(SORT_BY_ALIGNMENT(.noinit) SORT_BY_ALIGNMENT(.noinit.*))
 .noinit."WEST_TOPDIR/zephyr/drivers/wifi/esp32/src/esp_wifi_drv.c".3
                0x000000003d800010     0x1000 zephyr/drivers/wifi/libdrivers__wifi.a(esp_wifi_drv.c.obj)
                0x000000003d800010                esp_wifi_event_stack
 .noinit."WEST_TOPDIR/zephyr/drivers/wifi/esp32/src/esp_wifi_drv.c".2
                0x000000003d801010      0xb90 zephyr/drivers/wifi/libdrivers__wifi.a(esp_wifi_drv.c.obj)
 *libsubsys__net__l2__ethernet.a:(SORT_BY_ALIGNMENT(.noinit) SORT_BY_ALIGNMENT(.noinit.*))
 *libsubsys__net__lib__config.a:(SORT_BY_ALIGNMENT(.noinit) SORT_BY_ALIGNMENT(.noinit.*))
 *libsubsys__net__ip.a:(SORT_BY_ALIGNMENT(.noinit) SORT_BY_ALIGNMENT(.noinit.*))
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_mgmt.c".0
                0x000000003d801ba0      0x800 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_mgmt.c.obj)
                0x000000003d801ba0                mgmt_stack
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_tc.c".1
                0x000000003d8023a0     0x1000 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_tc.c.obj)
                0x000000003d8023a0                rx_stack
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/tcp.c".0
                0x000000003d8033a0      0x800 zephyr/subsys/net/ip/libsubsys__net__ip.a(tcp.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_mgmt.c".1
                0x000000003d803ba0      0x154 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_mgmt.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_pkt.c".3
                0x000000003d803cf4    0x20000 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_pkt.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_pkt.c".2
                0x000000003d823cf4      0xe00 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_pkt.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_pkt.c".1
                0x000000003d824af4    0x20000 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_pkt.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_pkt.c".0
                0x000000003d844af4      0xe00 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_pkt.c.obj)
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_pkt.c".k_mem_slab_buf_tx_pkts
                0x000000003d8458f4     0x2000 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_pkt.c.obj)
                0x000000003d8458f4                _k_mem_slab_buf_tx_pkts
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/net_pkt.c".k_mem_slab_buf_rx_pkts
                0x000000003d8478f4     0x2000 zephyr/subsys/net/ip/libsubsys__net__ip.a(net_pkt.c.obj)
                0x000000003d8478f4                _k_mem_slab_buf_rx_pkts
 .noinit."WEST_TOPDIR/zephyr/subsys/net/ip/tcp.c".k_mem_slab_buf_tcp_conns_slab
                0x000000003d8498f4     0x2d00 zephyr/subsys/net/ip/libsubsys__net__ip.a(tcp.c.obj)
 *libsubsys__net.a:(SORT_BY_ALIGNMENT(.noinit) SORT_BY_ALIGNMENT(.noinit.*))

And here is the system heap definition from hal_espressif/zephyr/esp_shared/src/common/heap_caps.c

#if defined(CONFIG_ESP_SPIRAM)
EXT_RAM_ATTR int _spiram_data_start;
STRUCT_SECTION_ITERABLE(k_heap, _spiram_heap) = {
    .heap = {
        .init_mem = &_spiram_data_start,
#if (CONFIG_ESP_SPIRAM_SIZE <= 0x400000)
        .init_bytes = CONFIG_ESP_SPIRAM_SIZE,
#else
        .init_bytes = 0x400000,
#endif
    },
};

This is looking like a bug or at least a conflict between the SPIRAM heap and CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM.

Also, what is the rational of only initialising the first 0x4000000 byes (4 MB) of memory?

I need to get this fixed to get back on track with the project (and also get back to PRs that I have been ignoring while getting this fixed). Any ideas on a quick workaround? Maybe move the heap to later in the map file and reduce the size? Will probably need to do the same for the network heap.

[sylvioalves]

@EricNRS, seems CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM is not working as expected. Do you really need that right now? Or you can continue without it?

[EricNRS]

Hi @sylvioalves, thanks for the reply. I do need it since I otherwise overflow dram0 whenever I enable WiFi and other functionality.

I have done some changes to move the SPIRAM heap from the beginning of SPIRAM if CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM is enabled. This passes my tests and I haven't seen any stability problems.

#if defined(CONFIG_ESP_SPIRAM)
EXT_RAM_ATTR int _spiram_data_start;

STRUCT_SECTION_ITERABLE(k_heap, _spiram_heap) = {
    .heap = {
#if defined(CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM)
        /* DEBUG - shift the SPIRAM heap up to half-way through memory to avoid conflict with WIFI SPIRAM usage */
        .init_mem = &_spiram_data_start + CONFIG_ESP_SPIRAM_SIZE / (2 * sizeof(int)),
        .init_bytes = CONFIG_ESP_SPIRAM_SIZE / 2,
#else /* CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM not defined, use all of SPIRAM for heap */
        .init_mem = &_spiram_data_start,
#if (CONFIG_ESP_SPIRAM_SIZE <= 0x400000)
        .init_bytes = CONFIG_ESP_SPIRAM_SIZE,
#else
        .init_bytes = 0x400000,
#endif /* (CONFIG_ESP_SPIRAM_SIZE <= 0x400000) */
#endif /* defined(CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM) */
    },
};
#endif

I am working on some changes to default.ld to use the entire remainder of SPIRAM for the heap. What I don't know is if the WiFi SPIRAM usage is limited to what is in the linker file (shown below) or if it dynamically uses more memory.

#define EXT_RAM_END 0x3E000000
#define EXT_RAM_ORG (EXT_RAM_END - CONFIG_ESP_SPIRAM_SIZE)
// . . .

  .ext_ram_noinit (NOLOAD) :
  {
#if defined(CONFIG_ESP32_WIFI_NET_ALLOC_SPIRAM)
    *libdrivers__wifi.a:(.noinit .noinit.*)
    *libsubsys__net__l2__ethernet.a:(.noinit .noinit.*)
    *libsubsys__net__lib__config.a:(.noinit .noinit.*)
    *libsubsys__net__ip.a:(.noinit .noinit.*)
    *libsubsys__net.a:(.noinit .noinit.*)
#endif
    _spiram_heap_start = ABSOLUTE(.);    <-- will this work for the SPIRAM heap or do the wifi drivers, etc dynamically use more memory than the link-time size being picked up here?
    _spiram_heap_end = ABSOLUTE(EXT_RAM_END);
  } > ext_ram_seg
#endif

[EricNRS]

OK, I have a fix for this and will upload after taking care of my other PR's that have some questions on them. We will have to see if these fixes should get included now or after the SOC cleanup. The heap code seems more complicated that necessary, so I wonder if it has evolved over time and is in need of a cleanup.

[EricNRS]

Opened #61359 for this.

[EricNRS]

Issued fixed.