Transient errors building baseline design (sky130hd/ibex) under local build

[CooperDeng]

Hi all,

I’ve been running into some transient issues while trying to run a baseline design using the following command:

make DESIGN_CONFIG=./designs/sky130hd/ibex/config.mk

The errors occur at seemingly random steps, and each run can fail differently. Here are some examples of the messages I get:

$ make DESIGN_CONFIG=./designs/sky130hd/ibex/config.mk 
...
  File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 153, in need_more_tokens
    if self.next_possible_simple_key() == self.tokens_taken:
  File "/usr/lib/python3/dist-packages/yaml/scanner.py", line 273, in next_possible_simple_key
    for level in self.possible_simple_keys:
RuntimeError: dictionary changed size during iteration
Command exited with non-zero status 1
Elapsed time: 0:00.05[h:]min:sec. CPU time: user 0.03 sys 0.01 (81%). Peak memory: 26520KB.
make[1]: *** [Makefile:454: do-yosys-canonicalize] Error 1
make: *** [Makefile:457: results/sky130hd/ibex/base/1_synth.rtlil] Error 2

$ make DESIGN_CONFIG=./designs/sky130hd/ibex/config.mk 
...
Setting clock period to 10.0
5. Executing HIERARCHY pass (managing design hierarchy).
5.1. Analyzing design hierarchy..
5.2. Analyzing design hierarchy..
6. Executing SYNTH pass.
6.1. Executing HIERARCHY pass (managing design hierarchy).
6.1.1. Analyzing design hierarchy..
6.1.2. Analyzing design hierarchy..
6.2. Executing PROC pass (convert processes to netlists).
6.2.1. Executing PROC_CLEAN pass (remove empty switches from decision trees).
6.2.2. Executing PROC_RMDEAD pass (remove dead branches from decision trees).
6.2.3. Executing PROC_PRUNE pass (remove redundant assignments in processes).
6.2.4. Executing PROC_INIT pass (extract init attributes).
6.2.5. Executing PROC_ARST pass (detect async resets in processes).
6.2.6. Executing PROC_ROM pass (convert switches to ROMs).
6.2.7. Executing PROC_MUX pass (convert decision trees to multiplexers).
6.2.8. Executing PROC_DLATCH pass (convert process syncs to latches).
6.2.9. Executing PROC_DFF pass (convert process syncs to FFs).
6.2.10. Executing PROC_MEMWR pass (convert process memory writes to cells).
6.2.11. Executing PROC_CLEAN pass (remove empty switches from decision trees).
6.2.12. Executing OPT_EXPR pass (perform const folding).
free(): invalid next size (fast)
Command terminated by signal 6
Elapsed time: 0:00.30[h:]min:sec. CPU time: user 0.25 sys 0.01 (86%). Peak memory: 52944KB.
make[1]: *** [Makefile:448: do-yosys] Error 134
make: *** [Makefile:460: results/sky130hd/ibex/base/1_1_yosys.v] Error 2

$ make DESIGN_CONFIG=./designs/sky130hd/ibex/config.mk 
...
6.2.4. Executing PROC_INIT pass (extract init attributes).
6.2.5. Executing PROC_ARST pass (detect async resets in processes).
6.2.6. Executing PROC_ROM pass (convert switches to ROMs).
6.2.7. Executing PROC_MUX pass (convert decision trees to multiplexers).
6.2.8. Executing PROC_DLATCH pass (convert process syncs to latches).
6.2.9. Executing PROC_DFF pass (convert process syncs to FFs).
ERROR: uncaught exception during Yosys command invoked from TCL
Command exited with non-zero status 1
Elapsed time: 0:00.26[h:]min:sec. CPU time: user 0.24 sys 0.01 (98%). Peak memory: 53744KB.
make[1]: *** [Makefile:448: do-yosys] Error 1

Each time I run the flow, it fails at a different step—sometimes during YAML parsing, sometimes deep in the Yosys passes. The behavior is completely non-deterministic, which makes debugging hard.

Also worth noting: I'm currently running the build locally on Ubuntu 22.04, but I'm not sure if that's contributing to the issue. Is anyone else seeing more stability running this inside the Docker container instead of locally?

Has anyone else experienced this kind of issue? Could this be related to Python/YAML versions, Yosys install, or maybe a memory-related race condition?

Any suggestions appreciated!!

[maliberty]

@povik any suggestions?

[oharboe]

Try running it 1000x:

for i in {1..1000}; do python scripts/non_stage_variables.py synth > /dev/null 2>&1; done

[CooperDeng]

Thanks a lot for looking into this.

I am running python 3.10.12:

$ python3 --version
Python 3.10.12

PyYaml version 5.4.1:

$ python3 -c "import yaml; print(yaml.__version__)"
5.4.1

I don't really have python symlinked:

$ python --version
Command 'python' not found, did you mean:
  command 'python3' from deb python3
  command 'python' from deb python-is-python3

After running it 1000x:

$ for i in {1..1000}; do python3 scripts/non_stage_variables.py synth > /dev/null 2>&1; done
Aborted (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)
Segmentation fault (core dumped)

[oharboe]

This would be the "smoking gun" of problems with your hardware or configuration, so I think this issue can be closed as mystery solved.

[oharboe]

@maliberty Close as mystery solved?

[maliberty]

I don't see anything to fix in OR