Heap overflow detection improvement (IDFGH-15349)

[bryghtlabs-richard]

Is your feature request related to a problem?

Today, I found a piece of code had corrupted a heap-buffer. (This is problem is mine alone)

However, I was surprised to find that it overwrote it by 1.7MB before a thread on the other core noticed, and crashed.

From the hardware available, it should be possible to make the heap much more aggressive at detecting these, and sometimes detect use-after-free as well.

Describe the solution you'd like.

I'd like to suggest that the ESP-HEAP treat physical memory as an array of MMU pages, and that access to those MMU pages be enabled/disabled as needed.

Ex:

• On heap creation, access to the first page of the heap is enabled(and will stay enabled forever, also it must stay enabled if there are variables before it in the same page). Middle pages of the heap will be disabled. The last page of the heap must stay enabled if there are variables after it.
• Once an allocation extends to a new page(s), access to those pages will be enabled.
• When allocations are freed, if the coalesced area covers a whole page, access to that page will be disabled.
• Access to a disabled page will fault and trap the error. This is nice, because the error is synchronous - the corrupting thread is much more likely to trap within 1x MMU page, or about 64KB.

Describe alternatives you've considered.

I won't write any more bugs ever again :)

Additional context.

No response

[igrr]

One thing to note is that ESP chips don't have synchronous MMU traps implemented in hardware until ESP32-P4, and even there they are disabled by default for performance reasons. There is also only an MMU for Flash and PSRAM access, internal RAM and the peripherals are accessed directly.

ESP-IDF will implement on-demand paging once there is hardware available to support this (e.g. implementing RISC-V Sv32 MMU).

[o-marshmallow]

Hello @bryghtlabs-richard ,

Thanks for your suggestion, as @igrr said, we don't have a hardware MMU for the internal RAM.

Regarding your issue, in ESP-IDF we provide a heap corruption detection feature that can be enabled via the menuconfig: Component config -> Heap memory debugging -> Heap corruption detection. You can either enable light poisoning (CONFIG_HEAP_POISONING_LIGHT) or comprehensive (CONFIG_HEAP_POISONING_COMPREHENSIVE).

This feature is documented here: https://docs.espressif.com/projects/esp-idf/en/stable/esp32/api-reference/system/heap_debug.html#heap-corruption-detection

[bryghtlabs-richard]

Regarding your issue, in ESP-IDF we provide a heap corruption detection feature that can be enabled via the menuconfig: Component config -> Heap memory debugging -> Heap corruption detection. You can either enable light poisoning (CONFIG_HEAP_POISONING_LIGHT) or comprehensive (CONFIG_HEAP_POISONING_COMPREHENSIVE).

This was enabled during the crash I mentioned, but because it only runs during heap operations, it did not trap this buffer overflow until much later, when another thread ran in parallel and detected the corruption. By this point, it was more difficult to debug the cause. While useful, this technique also has no capability to catch use-after-free.

There is also only an MMU for Flash and PSRAM access, internal RAM and the peripherals are accessed directly.
One thing to note is that ESP chips don't have synchronous MMU traps implemented in hardware until ESP32-P4

Ah, I understand - so this idea could only be useful for the PSRAM heap.

Is there an asynchronous MMU invalid access trap available? If so, we could limit corruption from nearly unlimited to a handful of cycles past the violation - usually correct thread/stack at least.

[igrr]

Yes, enabling the asynchronous invalid access trap is possible.

We would need to make the heap component aware of the MMU (whether it is available for the specific heap region or not, and how to use it) which could lead to more complex code and more coupling between components, but this idea can be considered.

[filzek]

@bryghtlabs-richard what about do a post check and create an wrapper for the functions? I have build one so far and its good for free and control where the variables are, so this could be an start point for alternative follow allocations, take a look at
https://github.com/filzek/Esp-IDF-MemoryAllocationControl

so, what could be used it to check the canary bytes on every each allocation, so, it could add extra canary to the tail of it by create a second address bound, like
pointer head
pointer real_allocation
pointer tail

@igrr is that possible to register inline allocations in the current idf witht the freertos, we know that this is possible elsewhere, but I do not know if is possible to we do it in freertos at all, so, it its possible we can add this to the tracker and will be fleasible to run it quite fast and track head and tails to see why its overwrite and who the offender is (off course demand time or line check not in realtime)

[igrr]

to run it quite fast and track head and tails to see why its overwrite and who the offender is

If I understood your suggestion correctly:

• Overriding allocation functions is already possible. That's how tracking malloc/free/etc calls ("heap tracing") currently works.
• It's not necessary to override allocation functions to check the poisoned bytes — this is already done by default if comprehensive heap corruption detection is enabled.
• Regarding identifying the "offender":
  • If the out-of-bounds write overflows the buffer just a bit (e.g. by overwriting the byte immediately after the allocated block), we can assume that the code related to that buffer is responsible. If heap allocation tracing is enabled, we do in theory have the information about the code where each block was allocated from. However presently, IDF's heap component doesn't connect these two features — poisoning and tracing. This is indeed a useful, if complex, improvement we could do.
  • If the out-of-bounds write overflows the allocated buffer by more than the size of the poisoning region, only instrumentation techniques such as kASAN can help — at the expense of code size and RAM usage. This would also be a useful improvement, however the code size increase may be prohibitive for many applications. Also there is an issue with recompiling the libraries (libc, Wi-Fi and Bluetooth pre-compiled libraries) with instrumentation options enabled.
  • If the out-of-bounds write overflows the allocated buffer by a lot, then the suggestion raised in the OP (protecting whole MMU pages) can help generate a trap which will directly point to the offending code.