looking for details on hierarchical flow

[Cronus-38]

Hello, I'm trying to find additional documentation on a hierarchical flow (i.e. bottom-up flow). I want to run a sub-block through the Openroad flow, then take the resulting GDSII and use it as-is (via manual placement) within my top-level run. I found the BLOCKS variable, and I see that if it's set then the flow will look for the sub-block files:

ifdef BLOCKS
ifeq ($(MAKELEVEL),0)
  $(info [INFO][FLOW] Invoked hierarchical flow.)
  $(foreach block,$(BLOCKS),$(info Block ${block} needs to be hardened.))
endif
  $(foreach block,$(BLOCKS),$(eval BLOCK_LEFS += ./results/$(PLATFORM)/$(DESIGN_NICKNAME)_$(block)/$(FLOW_VARIANT)/${block}.lef))
  $(foreach block,$(BLOCKS),$(eval BLOCK_LIBS += ./results/$(PLATFORM)/$(DESIGN_NICKNAME)_$(block)/$(FLOW_VARIANT)/${block}.lib))
  $(foreach block,$(BLOCKS),$(eval BLOCK_GDS += ./results/$(PLATFORM)/$(DESIGN_NICKNAME)_$(block)/$(FLOW_VARIANT)/6_final.gds))
  $(foreach block,$(BLOCKS),$(eval BLOCK_CDL += ./results/$(PLATFORM)/$(DESIGN_NICKNAME)_$(block)/$(FLOW_VARIANT)/6_final.cdl))
  $(foreach block,$(BLOCKS),$(eval BLOCK_LOG_FOLDERS += ./logs/$(PLATFORM)/$(DESIGN_NICKNAME)_$(block)/$(FLOW_VARIANT)/))

But I don't understand the function of the MAKELEVEL variable, and when I run with BLOCKS=my_sub-block it tells me that the block needs to be hardened. Clearly I'm not doing things right.

Thanks,
Cronus

[rovinski]

I'm not intimately familiar with the hierarchical flow but just as a note, MAKELEVEL is a makefile built-in variable which changes automatically when Make invokes another Make. So MAKELEVEL is 0 when the top-most block is called, then when it calls Make on the children, each one will have MAKELEVEL as 1, if any of those invocations called Make again, MAKELEVEL would be 2, and so on.

So ifeq ($(MAKELEVEL),0) executes that section only if it's the top-most Make invocation (i.e. you're calling it on the design top-level). I believe the strategy is to call it on the top-level, and then Make will automatically recurse through the child blocks and execute Make on those, exporting them, and then incorporating them into the hierarchy above them.

One of the issues you might see is that Make automatically passes its environment from the parent to the child. There's lots of cases where that's helpful, but it's not helpful here. So there are some steps taken to ensure the environment is reset when it hands it to a child.

At least, that's what I understand. I haven't run it myself.

[Cronus-38]

Thanks for the reply! Good info.

[maliberty]

See https://github.com/The-OpenROAD-Project/OpenROAD-flow-scripts/blob/master/flow/designs/asap7/aes-block/config.mk for an example that you can also run.

[Cronus-38]

Thanks Matt, I was able to get the aes-block design working for both the nangate45 and asap7 platforms. Here are some additional notes for anyone who is trying to set this up in the future and runs across this thread (the asap7 platform is assumed in the example below):

1. Assume I have a designed named my_top, and there's a sub-block mult within my_top that I want to harden into a macro before doing top-level synthesis and P&R.

2. Create a file named ./designs/asap7/my_top/block.mk. Note that block.mk is the actual name of the file, regardless of the name of the sub-block you're trying to harden. So you'll have both a top-level config.mk file and a single block.mk file which controls the settings for all of the sub-blocks you want to harden before running the top-level design. The contents of block.mk should be something like this:

    export PLATFORM := asap7

    export VERILOG_FILES = ./designs/src/my_top/mult.v   ;# Assume here there's only a single verilog file for the 
    sub-block.  Of course, if the RTL is parameterized then you'll need to read all of the verilog files to ensure everything in the hierarchy is elaborated with the correct parameterization.

    export SDC_FILE      = ./designs/$(PLATFORM)/my_top/mult.sdc

    export CORE_UTILIZATION       = 40
    export CORE_ASPECT_RATIO      = 1
    export CORE_MARGIN            = 2
    export PLACE_DENSITY          = 0.75

    export PLACE_PINS_ARGS = -annealing
    export HAS_IO_CONSTRAINTS = 0

3. Put the block-level timing constraints in a file named ./designs/asap7/my_top/mult.sdc

4. Add the following settings to designs/asap7/my_top/config.mk

   export BLOCKS ?= mult ;# This is the module name found in the mult.v verilog file
   export SYNTH_HIERARCHICAL = 1
   export RTLMP_FLOW = 1 ;# Set this for automatic placement of the hardened sub-block, or do the following for manual placement
   export MACRO_PLACEMENT_TCL = ./place_macros.tcl ;# This script is where you can specify the x,y location of the hardened sub-block(s) within the top-level floorplan. Use the instance name for the sub-block, in this case "imult": place_macro -macro_name imult -location { }

5. Execute "make" just the same as you would for the non-hierarchical flow. This will build the sub-block(s) first, followed by the building of the top-level. Note that the DESIGN_NICKNAME created for a sub-block will be the name of the sub-block appended to the top-level design name (in this example, the DESIGN_NICKNAME of the sub-block will be my_top_mult, and its data will reside in ./results/asap7/my_top_mult). If the sub-block design has already been built previously, it will be used as-is without rebuilding it. To re-build the sub-block design my_top_mult from scratch you can do: “make clean_all DESIGN_NICKNAME=my_top_mult”. Note, however, that this will not remove all of the previously built sub-block files due to an apparent bug. This is the remove command which is issued by clean_all in this instance:

rm -f ./results/asap7/my_top_mult/base/my_top.lib ./results/asap7/my_top_mult/base/my_top.lef

But the sub-block files which were built previously still remain:

root@f1168528f935:/OpenROAD-flow-scripts/flow# ls -al results/asap7/my_top_mult/base/*
-rw-r--r--. 1 root root 353391 Dec 26 17:19 results/asap7/my_top_mult/base/mult.lef
-rw-r--r--. 1 root root 100719 Dec 26 17:19 results/asap7/my_top_mult/base/mult.lib

Therefore, to fully remove the previously built sub-block, do:

rm -r results/asap7/my_top_mult

Now when you run make, the sub-block will be re-built from scratch.

I hope this helps someone!

[hw794]

Hi, I am trying to do the same thing since I kept getting the "block need to be hardened" error. However, when I try to build as what u mentioned above and also Matt's example, I get the error of :

cp /OpenROAD-flow-scripts/flow/designs/asap7/aes-block/constraint.sdc results/asap7/aes-block/base/1_synth.sdc
make: *** No rule to make target 'designs/asap7/aes-block/aes_rcon/config.mk', needed by 'results/asap7/aes-block_aes_rcon/base/aes_rcon.lef'. Stop.

It seems like the makefile is trying to create a hierarchy to match the design hierarchy. How I bypass this error before is to create a sub-folder inside the aes-block to store the config file. But that will throw the "need to be hardened" error later. Do u know why this might be the case? Thanks.

[Cronus-38]

Hi hw794,

Just to be clear, the "block needs to be hardened" message isn't an error, per se. It's just an informational message. It's only an "error" in the sense that it shouldn't be issued when the sub-block has already been hardened. Regardless, it's safe to ignore it, since the Openroad flow will automatically use the .lef and .lib files if they're available (that is, assuming the sub-block was already built successfully). If it can't find the .lef and .lib files then it will build the sub-block from scratch.

As for the other issue, before I try to address the problem you're seeing (the message which ends in "Stop."), let me give some additional background.

There are two ways to implement the hierarchical flow: You can either 1) run the flow using a single invocation of "make", or 2) build the sub-block standalone (i.e. in its own directory) if you want to finish it before attempting a top-level build. If you simply execute "make DESIGN_CONFIG=./designs/asap7/aes-block/config.mk" then you'll run Flow 1 (that is, you'll execute a single run which first builds the sub-block, then builds the top-level, all in one shot). Let's call Flow 1 the "single-run flow". That flow only requires the setup I mentioned in my earlier post on this thread. Here are the files you'll need to have in the ./designs/asap7/aes-block/ directory (again, for aes-block this is already set up).:

autotuner.json  block.mk  config.mk  constraint.sdc  metadata-base-ok.json  rules-base.json

So that's the single-run flow.

If you want to instead do flow Flow 2, which is what I call the "standalone" flow, then you need to create a new directory to hold the setup data for the standalone sub-block build. Based on what you showed, I suspect that's what you're trying to do. Indeed, to run that flow you would first create a config.mk file for the sub-block. That config.mk file and other files associated with the sub-block (for example, constraint.sdc) need to be in a directory named ./designs/asap7/aes-block_aes_rcon, since later on you'll be invoking make for the top-level run, and at that point the flow scripts will look in ./results/asap7/aes-block_aes_rcon/base for the .lef and .lib files that were generated by the sub-block run.

During your run you're seeing this:

make: *** No rule to make target 'designs/asap7/aes-block/aes_rcon/config.mk', needed by 'results/asap7/aes-block_aes_rcon/base/aes_rcon.lef'. Stop.

Based on the above output it appears to me you're trying to point the flow to a config.mk file in the ./designs/asap7/aes-block/aes_rcon/ directory, but if you're trying to use the standalone flow then you should instead call that directory aes-block_aes_rcon and point your DESIGN_CONFIG variable to that location/file.

Hopefully that's all clear? It sounds complicated, but once you understand how it works it's really simple. I would recommend executing the single-run flow on aes-block first to confirm it's working before trying to build the sub-block standalone.

IMPORTANT NOTE FOR STANDALONE FLOW: Since the outputs of the sub-block run need to be discoverable by the top-level run, you should specify "aes-block_aes_rcon" as the nickname for the design (so the aes-block_aes_rcon directory gets created in ./results/asap7/ during the sub-block run). That is, in the file ./designs/asap7/aes-block_aes_rcon/config.mk you'll want to have:

DESIGN_NICKNAME := aes-block_aes_rcon
DESIGN_NAME := aes_rcon
...

Let me know if you get it working!

[hw794]

Hi Cronus-38,
Thank you so much for the help; I am now able to reproduce the case! I noticed that the AES-block design generates several aes_rcon and aes_sbox macros. I understand that for a single macro, we can change the macro size, I/O, and other settings by adjusting the configuration file. However, I am wondering if we can set different I/O constraints for multiple instances of the same macro based on their positions on the chip. For example, in this AES-block design, can we assign the four aes_rcon macros to the four corners of the die, with each macro having I/O directed only toward the center of the die?

[maliberty]

You need to uniquify them in that case as you would get different layouts. You generally can't rotate 90 degrees but if you can accept mirroring you can reduce the number of unique items.