Trouble Minimizing ASAN Errors with bugpoint Due to Output in stderr

[sdjasj]

I've been using bugpoint recently to minimize ASAN-related bugs, but I've run into some trouble. Specifically, I have an input file that triggers a heap-use-after-free in Yosys, but when I try to use bugpoint, I can't capture the heap-use-after-free string using -grep because it appears in the standard error stream. As a result, I'm unable to minimize the code. Does anyone know how to minimize such errors?

[georgerennie]

One hacky approach could be to create the following bash script that just runs yosys with redirected stderr like

#! /usr/bin/env bash
yosys $@ 2&>1

Make sure this file is executable, then in bugpoint pass -yosys stderr_yosys.sh.

Maybe we should also make bugpoint grep over stderr as well as stdout by default too

[sdjasj]

@sdjasj did the stderr_yosys.sh work for you?

From your explanation, it seems the issue wasn't actually resolved... At the time, I used stderr_yosys.sh to get bugpoint to run successfully, but it showed the message: ERROR: The provided script file or command and Yosys binary do not crash on this design!. I assumed the bug wasn't reproducible, so I didn't look into it further. But now it seems the problem was that the ASAN error wasn't being captured in the log.

@KrystalDelusion I'm thinking about continuing to debug this bug—do you know of any way to make bugpoint catch ASAN errors? :)

[KrystalDelusion]

I'm working on it at the moment in #5068, but didn't have any examples for ASAN that were reproducible from RTLIL that I could test with; they were mostly related to read_verilog or AST parsing. If you have any current examples of ASAN or UBSAN errors that you can reproduce from RTLIL (not verilog) that you can share (even if they're not minimised) I would appreciate it 🙂

It may also work as-is with the two new options already available in the PR, but like I said I haven't tested with SAN errors.

[sdjasj]

@KrystalDelusion Of course! I'm happy to share this ASAN error. I've minimized the problematic design into RTLIL format and it reproduces reliably on my end. Since the RTLIL file is around 200 lines, I’ve uploaded it as an attachment: https://github.com/user-attachments/files/20186332/min.txt.

You can reproduce the issue with the following command:

read_rtlil min.txt
synth_greenpak4

On my setup, this yields the following output:

<suppressed ~2 debug messages>
=================================================================
==3959561==ERROR: AddressSanitizer: heap-use-after-free on address 0x610000090d7c at pc 0x55b1998bdc9a bp 0x7ffdfb7650b0 sp 0x7ffdfb7650a8
READ of size 4 at 0x610000090d7c thread T0
    #0 0x55b1998bdc99 in Yosys::hashlib::pool<Yosys::RTLIL::Cell*, Yosys::hashlib::hash_ops<Yosys::RTLIL::Cell*> >::do_hash(Yosys::RTLIL::Cell* const&) const (/usr/local/bin/yosys+0x21a0c99) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #1 0x55b1998be12f in Yosys::hashlib::pool<Yosys::RTLIL::Cell*, Yosys::hashlib::hash_ops<Yosys::RTLIL::Cell*> >::do_rehash() (/usr/local/bin/yosys+0x21a112f) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #2 0x55b1998bdd85 in Yosys::hashlib::pool<Yosys::RTLIL::Cell*, Yosys::hashlib::hash_ops<Yosys::RTLIL::Cell*> >::do_lookup(Yosys::RTLIL::Cell* const&, unsigned int&) const (/usr/local/bin/yosys+0x21a0d85) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #3 0x55b19a6a84ed in Yosys::hashlib::pool<Yosys::RTLIL::Cell*, Yosys::hashlib::hash_ops<Yosys::RTLIL::Cell*> >::insert(Yosys::RTLIL::Cell*&&) (/usr/local/bin/yosys+0x2f8b4ed) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #4 0x55b19ad9d660 in (anonymous namespace)::NlutmapPass::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x3680660) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #5 0x55b199534079 in Yosys::Pass::call(Yosys::RTLIL::Design*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) (/usr/local/bin/yosys+0x1e17079) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #6 0x55b19953374f in Yosys::Pass::call(Yosys::RTLIL::Design*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (/usr/local/bin/yosys+0x1e1674f) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #7 0x55b19975bd46 in Yosys::shell(Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x203ed46) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #8 0x55b19948b699 in main (/usr/local/bin/yosys+0x1d6e699) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #9 0x7f4395d55d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
    #10 0x7f4395d55e3f in __libc_start_main csu/../csu/libc-start.c:392:3
    #11 0x55b1993c1a74 in _start (/usr/local/bin/yosys+0x1ca4a74) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)

0x610000090d7c is located 60 bytes inside of 192-byte region [0x610000090d40,0x610000090e00)
freed by thread T0 here:
    #0 0x55b19947feed in operator delete(void*) (/usr/local/bin/yosys+0x1d62eed) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #1 0x55b1995f258d in Yosys::RTLIL::Module::remove(Yosys::RTLIL::Cell*) (/usr/local/bin/yosys+0x1ed558d) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #2 0x55b19a59d264 in (anonymous namespace)::rmunused_module_cells(Yosys::RTLIL::Module*, bool) (/usr/local/bin/yosys+0x2e80264) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #3 0x55b19a598da1 in (anonymous namespace)::rmunused_module(Yosys::RTLIL::Module*, bool, bool, bool) (/usr/local/bin/yosys+0x2e7bda1) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #4 0x55b19a59758d in (anonymous namespace)::OptCleanPass::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x2e7a58d) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #5 0x55b199534079 in Yosys::Pass::call(Yosys::RTLIL::Design*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) (/usr/local/bin/yosys+0x1e17079) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #6 0x55b19953374f in Yosys::Pass::call(Yosys::RTLIL::Design*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (/usr/local/bin/yosys+0x1e1674f) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #7 0x55b199534f8d in Yosys::Pass::call_on_module(Yosys::RTLIL::Design*, Yosys::RTLIL::Module*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (/usr/local/bin/yosys+0x1e17f8d) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #8 0x55b19ad9e8ba in (anonymous namespace)::NlutmapWorker::run_abc(int) (/usr/local/bin/yosys+0x36818ba) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #9 0x55b19ad9c262 in (anonymous namespace)::NlutmapPass::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x367f262) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #10 0x55b199534079 in Yosys::Pass::call(Yosys::RTLIL::Design*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) (/usr/local/bin/yosys+0x1e17079) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #11 0x55b19953374f in Yosys::Pass::call(Yosys::RTLIL::Design*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (/usr/local/bin/yosys+0x1e1674f) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #12 0x55b19975bd46 in Yosys::shell(Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x203ed46) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #13 0x55b19948b699 in main (/usr/local/bin/yosys+0x1d6e699) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #14 0x7f4395d55d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

previously allocated by thread T0 here:
    #0 0x55b19947f68d in operator new(unsigned long) (/usr/local/bin/yosys+0x1d6268d) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #1 0x55b1995f8174 in Yosys::RTLIL::Module::addCell(Yosys::RTLIL::IdString, Yosys::RTLIL::IdString) (/usr/local/bin/yosys+0x1edb174) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #2 0x55b19abdcae8 in (anonymous namespace)::abc_module(Yosys::RTLIL::Design*, Yosys::RTLIL::Module*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, std::vector<int, std::allocator<int> >, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, std::vector<Yosys::RTLIL::Cell*, std::allocator<Yosys::RTLIL::Cell*> > const&, bool, bool, bool, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >&) (/usr/local/bin/yosys+0x34bfae8) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #3 0x55b19abc3c13 in (anonymous namespace)::AbcPass::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x34a6c13) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #4 0x55b199534079 in Yosys::Pass::call(Yosys::RTLIL::Design*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) (/usr/local/bin/yosys+0x1e17079) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #5 0x55b19953374f in Yosys::Pass::call(Yosys::RTLIL::Design*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (/usr/local/bin/yosys+0x1e1674f) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #6 0x55b19953471e in Yosys::Pass::call_on_selection(Yosys::RTLIL::Design*, Yosys::RTLIL::Selection const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (/usr/local/bin/yosys+0x1e1771e) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #7 0x55b19ad9e657 in (anonymous namespace)::NlutmapWorker::run_abc(int) (/usr/local/bin/yosys+0x3681657) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #8 0x55b19ad9c262 in (anonymous namespace)::NlutmapPass::execute(std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x367f262) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #9 0x55b199534079 in Yosys::Pass::call(Yosys::RTLIL::Design*, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >) (/usr/local/bin/yosys+0x1e17079) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #10 0x55b19953374f in Yosys::Pass::call(Yosys::RTLIL::Design*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (/usr/local/bin/yosys+0x1e1674f) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #11 0x55b19975bd46 in Yosys::shell(Yosys::RTLIL::Design*) (/usr/local/bin/yosys+0x203ed46) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #12 0x55b19948b699 in main (/usr/local/bin/yosys+0x1d6e699) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e)
    #13 0x7f4395d55d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16

SUMMARY: AddressSanitizer: heap-use-after-free (/usr/local/bin/yosys+0x21a0c99) (BuildId: a060db898e8105d4f113ab9bcdcd75a50299f22e) in Yosys::hashlib::pool<Yosys::RTLIL::Cell*, Yosys::hashlib::hash_ops<Yosys::RTLIL::Cell*> >::do_hash(Yosys::RTLIL::Cell* const&) const
Shadow bytes around the buggy address:
  0x0c208000a150: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c208000a160: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c208000a170: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c208000a180: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c208000a190: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
=>0x0c208000a1a0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd[fd]
  0x0c208000a1b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c208000a1c0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c208000a1d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x0c208000a1e0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
  0x0c208000a1f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==3959561==ABORTING

During local debugging, I managed to isolate the issue by reducing the synth_greenpak4 flow to the following sequence:

read_verilog -lib +/greenpak4/cells_sim.v
hierarchy -check -auto-top
proc
flatten
tribuf -logic
synth -run coarse
extract_counter -pout GP_DCMP,GP_DAC -maxwidth 14
opt -fast -mux_undef -undriven -fine
memory_map
opt -undriven -fine
techmap -map +/techmap.v -map +/greenpak4/cells_latch.v
dfflibmap -prepare -liberty +/greenpak4/gp_dff.lib
opt -fast -noclkinv -noff
nlutmap -assert -luts 2,8,16,2

The crash occurs specifically in the last step:

nlutmap -assert -luts 2,8,16,2

[KrystalDelusion]

That's perfect, minimized to

# Generated by Yosys 0.53+24 (git sha1 65a5227c1, ccache clang++ 18.1.8 -fPIC -O3 -O1 -fno-omit-frame-pointer -fno-optimize-sibling-calls -fsanitize=address)
autoidx 78
attribute \cells_not_processed 1
attribute \src "test.v:1.1-63.10"
module \a
  attribute $bugpoint 1
  wire output 2 $auto$bugpoint.cc:300:simplify_something$27
  attribute $bugpoint 1
  wire output 3 $auto$bugpoint.cc:300:simplify_something$37
  attribute \src "test.v:35.16-35.28"
  wire width 18 $mul$test.v:35$1_Y
  attribute \src "test.v:13.18-13.19"
  wire width 18 offset 8 input 1 \b
  attribute \src "test.v:35.16-35.28"
  cell $mul $mul$test.v:35$1
    parameter \A_SIGNED 0
    parameter \A_WIDTH 18
    parameter \B_SIGNED 0
    parameter \B_WIDTH 12
    parameter \Y_WIDTH 18
    connect \A \b
    connect \B 12'111100001110
    connect \Y $mul$test.v:35$1_Y
  end
  attribute \src "test.v:35.16-41.20"
  cell $reduce_bool $reduce_bool$test.v:35$4
    parameter \A_SIGNED 0
    parameter \A_WIDTH 18
    parameter \Y_WIDTH 1
    connect \A $mul$test.v:35$1_Y
    connect \Y $auto$bugpoint.cc:300:simplify_something$37
  end
end

with

read_rtlil min.txt
bugpoint -yosys ./yosys -command synth_greenpak4 -err_grep "heap-use-after-free" -clean
write_rtlil min_asan.il

on 65a5227, which means -err_grep option does work for SAN :)

[sdjasj]

which means -err_grep option does work for SAN :)

This is very helpful for testers! Thank you for adding this feature :)