Random seg faults at running setup.sh

[CooperDeng]

Hello gang,

I am trying to build the flow script locally on ubuntu 22.04.

Similar to the post I have made in here, I am trying to re-build the flow script, and running into some random segmentation faults in the process:

$ sudo ./setup.sh 
Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease
Hit:2 http://ca.archive.ubuntu.com/ubuntu jammy InRelease                      
Hit:3 http://ca.archive.ubuntu.com/ubuntu jammy-updates InRelease              
Hit:4 http://ca.archive.ubuntu.com/ubuntu jammy-backports InRelease            
Hit:5 https://dl.google.com/linux/chrome/deb stable InRelease                  
Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease               
Hit:7 https://ppa.launchpadcontent.net/graphics-drivers/ppa/ubuntu jammy InRelease
Hit:8 https://ppa.launchpadcontent.net/yktooo/ppa/ubuntu jammy InRelease
./tools/OpenROAD/etc/DependencyInstaller.sh: line 334:  9155 Segmentation fault      (core dumped) apt-get -y update

$ sudo ./setup.sh
...
> g++ -x c++ -std=c++11 -pthread -O2 -s -DNDEBUG bindjam.cpp builtins.cpp class.cpp command.cpp compile.cpp constants.cpp cwd.cpp debug.cpp debugger.cpp events.cpp execcmd.cpp execnt.cpp execunix.cpp filent.cpp filesys.cpp fileunix.cpp frames.cpp function.cpp glob.cpp hash.cpp hcache.cpp hdrmacro.cpp headers.cpp jam_strings.cpp jam.cpp jamgram.cpp lists.cpp make.cpp make1.cpp md5.cpp mem.cpp modules.cpp native.cpp option.cpp output.cpp parse.cpp pathnt.cpp pathsys.cpp pathunix.cpp regexp.cpp rules.cpp scan.cpp search.cpp startup.cpp tasks.cpp timestamp.cpp value.cpp variable.cpp w32_getreg.cpp mod_command_db.cpp mod_db.cpp mod_jam_builtin.cpp mod_jam_class.cpp mod_jam_errors.cpp mod_jam_modules.cpp mod_order.cpp mod_path.cpp mod_property_set.cpp mod_regex.cpp mod_sequence.cpp mod_set.cpp mod_string.cpp mod_summary.cpp mod_sysinfo.cpp mod_version.cpp -o b2
tools/build/src/engine/b2
Unicode/ICU support for Boost.Regex?... /usr
Generating B2 configuration in project-config.jam for gcc...

Bootstrapping is done. To build, run:

    ./b2
    
To generate header files, run:

    ./b2 headers

The configuration generated uses gcc to build by default. If that is
unintended either use the --with-toolset option or adjust configuration, by
editing 'project-config.jam'.

Further information:

   - Command line help:
     ./b2 --help
     
   - Getting started guide: 
     http://www.boost.org/more/getting_started/unix-variants.html
     
   - B2 documentation:
     http://www.boost.org/build/

Performing configuration checks

    - default address-model    : 64-bit [1]
    - default architecture     : x86 [1]
    - zlib                     : yes [2]
    - bzip2                    : no [2]
    - lzma                     : yes [2]
    - zstd                     : no [2]
    - has_lzma_cputhreads builds : yes [2]
    - BOOST_COMP_GNUC >= 4.3.0 : yes [2]
    - BOOST_COMP_GNUC >= 4.3.0 : yes [3]
    - std_wstreambuf builds    : yes [2]
    - std_wstreambuf           : yes [2]
./tools/OpenROAD/etc/DependencyInstaller.sh: line 52:  8157 Segmentation fault      (core dumped) ./b2 install --with-iostreams --with-test --with-serialization --with-system --with-thread -j $(nproc)

I thought this has something do with my hardware, specifically my ram, so I tried everything - undervolting, underclocking, running memtest86 every time I modified the ram behaviour (total time running mem86 would be around 10-15 hours), and didn't find anything wrong with my ram.

I also have tried turning off the Intel XMP overclocking on my cpu, as well as undervolting it a bit, but still no result.

Any insights on where this could be coming from? I have spent a good amount of time configuring my hardwares for the past week, if not 2 weeks.

Thanks a lot!

[maliberty]

Perhaps you have a bad compiler installation? Another path is to use docker.

[CooperDeng]

Tried docker installation:

$ ./build_openroad.sh
21 69.21 make[1]: Leaving directory '/OpenROAD-flow-scripts/tools/yosys/abc'
#21 69.22 [ 99%] Building yosys-abc
#21 69.80 PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
#21 69.80 Stack dump:
#21 69.80 0.	Program arguments: clang++ -o passes/opt/opt_expr.o -c -D__TIME__=0 -D__DATE__=0 -D__TIMESTAMP__=0 -Wno-builtin-macro-redefined -Wall -Wextra -ggdb -I. -I./ -MD -MP -D_YOSYS_ -fPIC -I/OpenROAD-flow-scripts/tools/install/yosys/include -DYOSYS_VER=\\0.51\\ -DYOSYS_MAJOR=0 -DYOSYS_MINOR=51 -DYOSYS_COMMIT=0.51 -std=c++17 -O3 -DYOSYS_ENABLE_READLINE -DYOSYS_ENABLE_PLUGINS -DYOSYS_ENABLE_GLOB -DYOSYS_ENABLE_ZLIB -I/usr/include/tcl8.6 -DYOSYS_ENABLE_TCL -DYOSYS_ENABLE_ABC -DYOSYS_ENABLE_COVER passes/opt/opt_expr.cc
#21 69.80 1.	<eof> parser at end of file
#21 69.80 2.	Code generation
#21 69.80 3.	Running pass 'Function Pass Manager' on module 'passes/opt/opt_expr.cc'.
#21 69.80 4.	Running pass 'Live DEBUG_VALUE analysis' on function '@_ZNSt6vectorIN5Yosys7hashlib4dictINS0_5RTLIL8IdStringENS0_8CellTypeENS1_8hash_opsIS4_EEE7entry_tESaIS9_EED2Ev'
#21 69.80 Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var LLVM_SYMBOLIZER_PATH to point to it):
#21 69.80 /lib/x86_64-linux-gnu/libLLVM-14.so.1(_ZN4llvm3sys15PrintStackTraceERNS_11raw_ostreamEi+0x31)[0x7e702ffc0d01]
#21 69.81 /lib/x86_64-linux-gnu/libLLVM-14.so.1(_ZN4llvm3sys17RunSignalHandlersEv+0xee)[0x7e702ffbea3e]
#21 69.81 /lib/x86_64-linux-gnu/libLLVM-14.so.1(_ZN4llvm3sys15CleanupOnSignalEm+0xfb)[0x7e702ffc00ab]
#21 69.81 /lib/x86_64-linux-gnu/libLLVM-14.so.1(+0xd6bdff)[0x7e702feecdff]
#21 69.81 /lib/x86_64-linux-gnu/libc.so.6(+0x42520)[0x7e702ec67520]
#21 69.81 clang: error: clang frontend command failed with exit code 139 (use -v to see invocation)
#21 69.81 Ubuntu clang version 14.0.0-1ubuntu1.1
#21 69.81 Target: x86_64-pc-linux-gnu
#21 69.81 Thread model: posix
#21 69.81 InstalledDir: /usr/bin
#21 69.86 clang: note: diagnostic msg: 
#21 69.86 ********************
#21 69.86 
#21 69.86 PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
#21 69.86 Preprocessed source(s) and associated run script(s) are located at:
#21 69.86 clang: note: diagnostic msg: /tmp/opt_expr-fc09c0.cpp
#21 69.86 clang: note: diagnostic msg: /tmp/opt_expr-fc09c0.sh
#21 69.86 clang: note: diagnostic msg: 
#21 69.86 
#21 69.86 ********************
#21 69.88 make: *** [Makefile:739: passes/opt/opt_expr.o] Error 139
#21 69.88 make: Leaving directory '/OpenROAD-flow-scripts/tools/yosys'
#21 ERROR: process "/bin/sh -c echo \"\" > tools/yosys/abc/.gitcommit &&   env CFLAGS=\"-D__TIME__=0 -D__DATE__=0 -D__TIMESTAMP__=0 -Wno-builtin-macro-redefined\"   CXXFLAGS=\"-D__TIME__=0 -D__DATE__=0 -D__TIMESTAMP__=0 -Wno-builtin-macro-redefined\"   ./build_openroad.sh --no_init --local --threads ${numThreads}" did not complete successfully: exit code: 2
------
 > [orfs-builder-base 2/2] RUN echo "" > tools/yosys/abc/.gitcommit &&   env CFLAGS="-D__TIME__=0 -D__DATE__=0 -D__TIMESTAMP__=0 -Wno-builtin-macro-redefined"   CXXFLAGS="-D__TIME__=0 -D__DATE__=0 -D__TIMESTAMP__=0 -Wno-builtin-macro-redefined"   ./build_openroad.sh --no_init --local --threads 32:
69.86 
69.86 PLEASE ATTACH THE FOLLOWING FILES TO THE BUG REPORT:
69.86 Preprocessed source(s) and associated run script(s) are located at:
69.86 clang: note: diagnostic msg: /tmp/opt_expr-fc09c0.cpp
69.86 clang: note: diagnostic msg: /tmp/opt_expr-fc09c0.sh
69.86 clang: note: diagnostic msg: 
69.86 
69.86 ********************
69.88 make: *** [Makefile:739: passes/opt/opt_expr.o] Error 139
69.88 make: Leaving directory '/OpenROAD-flow-scripts/tools/yosys'
------
ERROR: failed to solve: process "/bin/sh -c echo \"\" > tools/yosys/abc/.gitcommit &&   env CFLAGS=\"-D__TIME__=0 -D__DATE__=0 -D__TIMESTAMP__=0 -Wno-builtin-macro-redefined\"   CXXFLAGS=\"-D__TIME__=0 -D__DATE__=0 -D__TIMESTAMP__=0 -Wno-builtin-macro-redefined\"   ./build_openroad.sh --no_init --local --threads ${numThreads}" did not complete successfully: exit code: 2

Upon further tries with nothing modified it actually passes and successfully build:

$ ./build_openroad.sh
...
#21 222.2 -- Installing: /OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad
#21 222.2 -- Set runtime path of "/OpenROAD-flow-scripts/tools/install/OpenROAD/bin/openroad" to "/OpenROAD-flow-scripts/dependencies/lib:/OpenROAD-flow-scripts/dependencies/lib64:/opt/or-tools/lib"
#21 DONE 222.4s

#22 [stage-2 10/11] COPY --link --from=orfs-builder-base /OpenROAD-flow-scripts/tools/install tools/install
#22 DONE 0.1s

#23 [stage-2 11/11] COPY --link      --exclude=.git* --exclude=tools/ --exclude=docs/ --exclude=docker/      --exclude=flow/designs --exclude=flow/platforms --exclude=flow/scripts      --exclude=flow/test --exclude=flow/tutorials --exclude=flow/util   . ./
#23 DONE 0.1s

#24 exporting to image
#24 exporting layers
#24 exporting layers 1.0s done
#24 writing image sha256:f8c0e83e8bf46f6201c68f9b89513e9fd02a5e56b0d8faabcf6ef78d5230d6cd done
#24 naming to docker.io/openroad/flow-ubuntu22.04-builder:bc62c0 done
#24 DONE 1.0s

However, when trying to run some default designs, still getting hit with errors:

$ util/docker_shell make
...
Unable to find image 'openroad/flow-ubuntu22.04-builder:latest' locally
latest: Pulling from openroad/flow-ubuntu22.04-builder
6eae748569ee: Pull complete 
b304dd8a2331: Pull complete 
ab0b2001f7c2: Pull complete 
56f6e60563fd: Pull complete 
dfdcc530d846: Pull complete 
de31ef3d36f3: Pull complete 
bb3310e41db1: Pull complete 
4f4fb700ef54: Pull complete 
c320075afe4d: Pull complete 
Digest: sha256:5d4d20d3d5638fa5c79b05c6f9b240c7e5691bf8b6c47c9a46d9bb0e0d53bb09
Status: Downloaded newer image for openroad/flow-ubuntu22.04-builder:latest
+ mkdir /tmp/xdg-run
+ cd /OpenROAD-flow-scripts/flow
+ . ../env.sh
++ [[ linux-gnu == \d\a\r\w\i\n* ]]
++++ readlink -f ../env.sh
+++ dirname /OpenROAD-flow-scripts/env.sh
++ DIR=/OpenROAD-flow-scripts
++ '[' -f /opt/rh/rh-python38/enable ']'
++ export OPENROAD=/OpenROAD-flow-scripts/tools/OpenROAD
++ OPENROAD=/OpenROAD-flow-scripts/tools/OpenROAD
++ echo 'OPENROAD: /OpenROAD-flow-scripts/tools/OpenROAD'
OPENROAD: /OpenROAD-flow-scripts/tools/OpenROAD
++ export PATH=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH=/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH=/OpenROAD-flow-scripts/tools/install/LSOracle/bin:/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/OpenROAD-flow-scripts/tools/install/LSOracle/bin:/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ export PATH=/OpenROAD-flow-scripts/dependencies/bin:/OpenROAD-flow-scripts/tools/install/LSOracle/bin:/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ PATH=/OpenROAD-flow-scripts/dependencies/bin:/OpenROAD-flow-scripts/tools/install/LSOracle/bin:/OpenROAD-flow-scripts/tools/install/yosys/bin:/OpenROAD-flow-scripts/tools/install/OpenROAD/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
++ [[ linux-gnu == \d\a\r\w\i\n* ]]
++ export FLOW_HOME=/OpenROAD-flow-scripts/flow
++ FLOW_HOME=/OpenROAD-flow-scripts/flow
+ make
/OpenROAD-flow-scripts/flow/scripts/flow.sh 5_1_grt global_route
Running global_route.tcl, stage 5_1_grt
[INFO ORD-0030] Using 32 thread(s).
Error: global_route.tcl, 4 incompatible database schema revision 0.98 > 0.84
Command exited with non-zero status 1
Elapsed time: 0:00.16[h:]min:sec. CPU time: user 0.14 sys 0.01 (100%). Peak memory: 92624KB.
make[1]: *** [Makefile:554: do-5_1_grt] Error 1
make: *** [Makefile:554: results/nangate45/gcd/base/5_1_grt.odb] Error 2

Apologies for the terminal dump, but what I have done were:
Suspected hardware instability due to:
14th-gen Intel CPU, 4x DDR5 DIMMs (common instability issues)

Tweaked BIOS settings:

• Undervolted CPU
• Undervolted and underclocked RAM

Ran MemTest86 every single time I made changes in BIOS, leaving overnight and returned no errors.

Stress-tested GCC with C programs that malloc/free memory randomly — no issues found

Suspected NVIDIA proprietary driver interference:

• Reinstalled NVIDIA drivers
• Reinstalling GCC afterwards crashed my ubuntu GUI somehow, but it's fixed by reinstalling gdm3

Completely out of ideas on where it can actually go wrong, any suggestions would be greatly appreciated!

[maliberty]

To use docker see https://openroad-flow-scripts.readthedocs.io/en/latest/user/BuildWithDocker.html

[CooperDeng]

Weird stuff, so I am able to build the docker image and successfully entered, inside docker:

/OpenROAD-flow-scripts# source ./env.sh 
OPENROAD: /OpenROAD-flow-scripts/tools/OpenROAD
/OpenROAD-flow-scripts# yosys -help

Usage: yosys [options] [<infile> [..]]

    -Q
        suppress printing of banner (copyright, disclaimer, version)

    -T
        suppress printing of footer (log hash, version, timing statistics)

    -q
        quiet operation. only write warnings and error messages to console
        use this option twice to also quiet warning messages

...

/OpenROAD-flow-scripts# openroad -help
Usage: openroad [-help] [-version] [-no_init] [-no_splash] [-exit] [-gui] [-threads count|max] [-log file_name] [-metrics file_name] cmd_file
  -help                 show help and exit
  -version              show version and exit
  -no_init              do not read .openroad init file
  -threads count|max    use count threads
  -no_splash            do not show the license splash at startup
  -exit                 exit after reading cmd_file
  -gui                  start in gui mode
  -python               start with python interpreter [limited to db operations]
  -log <file_name>      write a log in <file_name>
  -metrics <file_name>  write metrics in <file_name> in JSON format
  cmd_file              source cmd_file

/OpenROAD-flow-scripts# cd flow`
/OpenROAD-flow-scripts/flow# make
/OpenROAD-flow-scripts/flow/scripts/synth.sh /OpenROAD-flow-scripts/flow/scripts/synth_canonicalize.tcl ./logs/nangate45/gcd/base/1_1_yosys_canonicalize.log
1. Executing Verilog-2005 frontend: /OpenROAD-flow-scripts/flow/designs/src/gcd/gcd.v
2. Executing Liberty frontend: ./objects/nangate45/gcd/base/lib/NangateOpenCellLibrary_typical.lib
ERROR: TCL interpreter returned an error: Yosys command produced an error
Command exited with non-zero status 1
Elapsed time: 0:00.08[h:]min:sec. CPU time: user 0.07 sys 0.01 (100%). Peak memory: 31744KB.
make[1]: *** [Makefile:291: do-yosys-canonicalize] Error 1
make: *** [Makefile:294: results/nangate45/gcd/base/1_synth.rtlil] Error 2

Ran make inside of flow multiple times:

/OpenROAD-flow-scripts/flow# make
m/OpenROAD-flow-scripts/flow/scripts/synth.sh /OpenROAD-flow-scripts/flow/scripts/synth_canonicalize.tcl ./logs/nangate45/gcd/base/1_1_yosys_canonicalize.log
1. Executing Verilog-2005 frontend: /OpenROAD-flow-scripts/flow/designs/src/gcd/gcd.v
2. Executing Liberty frontend: ./objects/nangate45/gcd/base/lib/NangateOpenCellLibrary_typical.lib
ERROR: TCL interpreter returned an error: Yosys command produced an error
Command exited with non-zero status 1
Elapsed time: 0:00.13[h:]min:sec. CPU time: user 0.12 sys 0.00 (100%). Peak memory: 32000KB.
make[1]: *** [Makefile:291: do-yosys-canonicalize] Error 1
make: *** [Makefile:294: results/nangate45/gcd/base/1_synth.rtlil] Error 2

/OpenROAD-flow-scripts/flow# make
/OpenROAD-flow-scripts/flow/scripts/synth.sh /OpenROAD-flow-scripts/flow/scripts/synth_canonicalize.tcl ./logs/nangate45/gcd/base/1_1_yosys_canonicalize.log
1. Executing Verilog-2005 frontend: /OpenROAD-flow-scripts/flow/designs/src/gcd/gcd.v
2. Executing Liberty frontend: ./objects/nangate45/gcd/base/lib/NangateOpenCellLibrary_typical.lib
ERROR: TCL interpreter returned an error: Yosys command produced an error
Command exited with non-zero status 1
Elapsed time: 0:00.13[h:]min:sec. CPU time: user 0.12 sys 0.01 (99%). Peak memory: 31744KB.
make[1]: *** [Makefile:291: do-yosys-canonicalize] Error 1
make: *** [Makefile:294: results/nangate45/gcd/base/1_synth.rtlil] Error 2

I am so confused right now

[CooperDeng]

Aha! Well, it turns out, just like what @oharboe has mentioned from my previous post here, there IS something wrong with my hardware setup - one of my CPU's P-core is completely busted!

Took me long enough to figure it out, but recently even by just running

sudo apt-get -y update

would actually result in segmentation fualts for me. That shows something to be seriously wrong with my hardware setup.

Thankfully some random guy was having the same problem from 3 months ago , I was able to isolate and disable the corresponding core of my CPU by following the guide posted there.

After doing so my project shows no significant errors and built successfully locally, what a journey.

PS: I am running 14900kf on ASUS ROG STRIX Z790-H. Intel, never again!