Code Anatomy on Debug Module

[DecodeTheEncoded]

Debugging is a core concept in terms of soc design. RISC-V debug spec has supports for two forms of debugging. External debugging and native debugging.
External debugging is that the internal state of a hardware platform could be obtained or modified by an external debugger. You can read the Chapter 2 System Overview of RISC-V debug spec to obtain a holistic understanding of RISC-V debugging system. Specifically, a high level debugging intent is translated to a sequence of JTAG IR & DR scans. These JTAG IR&DR scans are handled inside the JTAG TAP module(JtagTapController). JTAG data register dmi is used as a proxy that can read from or write to the DMI address space of DM. Once the dmi jtag data register is selected (by scanning into IR the dmi data chain index 0x11, the value will take effect in Update-IR), we then can scan into the dmi jtag data register the corresponding value through a series of Shift-DRs. Once the needed value is shifted in, the DTM starts the operation specified in op(field of the dmi jtag data register) during jtag state Update-DR: sending translated dmi access request(op == 1 for read from addressand op == 2 for write data to address) through the DMI bus.
The DTM is connected with the DM via DMI bus, DM(debug module) is where magic of external debugging happens. One DM manages a bunch of harts in terms of external debugging(normally, there is only one DM in charging of all the harts in a hardware platform; And specific debugging operation can be applied to harts that are selected by hart array mask scheme hasel and hartsel fields in the dmcontrol dmi register);
The DM receives specific dmi register access request from DMI bus; For example, when there is a halt request from debugger to the DM(asserting haltreq in dmcontrol), DM will fire an interrupt to all the selected harts. Consequently the hart's execution flow will then jump to debug rom serviced by the debug module(trapToDebug), the hart then acknowledges its halt status to Debug Module by writing its hartid to a specific address(HALTED in RC, the simultaneous write to HALTED by multiple harts is serialized by the tilelink protocol), and waits for subsequent orders from debug module in a loop, DM is aware of value changes at HALTED(hartHaltedWrEn), and therefore knows the hart indicated by hartid stored at HALTED now is in halting status. The DM can then conduct specific debugging operations on that hart. This is what the dmi command register come to play.
The command can be used to access internal states of the hart, like GPRs, etc. Once a command is wrote by debugger(control signal goAbstract in RC impl is asserted), DM will construct some specific instructions at the address of ABSTRACT(0)(abstractGeneratedMem(0)) and ABSTRACT(1)(abstractGeneratedMem(1)) based on the detailed configuration of a command, for example:transfer of command being 0 means don't conduct the operation specified by write field, 1 means otherwise conduct the operation specified by write. So, if 0 is set, a nop will be put in ABSTRACT(0), while if 1 is set and the write field is asserted, the instruction at ABSTRACT(0) will be LW(write being true means write to register specified by regno, therefore it'a memory load), otherwise it's a SW(Copy data from the specified regno register into arg0 portion of data.), the dmi register data is the MR(message register), it's used for exchanging values between the hart being debugged and the debug module, for example, the LW(if any) instruction in ABSTRACT(0) means copy data from the arg0 portion of data into the specified GPR. The content at ABSTRACT(1) is decided by the postexec field of command, if it's asserted, that means execute the program in the Program Buffer exactly once after performing the transfer, if any. So, a nop will be inserted at ABSTRACT(1) when postexec is asserted, the execution flow will slip into the program buffer right next to the ABSTRACT(1). If postexec is 0, then this means no automatic execution of program buffer, therefore a ebreak should be inserted here, marking end of abstract command execution. It's worth noting that program buffer is a memory region right next to ABSTRACT(1), the debugger could put some instructions there to perform specific debugging operations. The program buffer scheme can be used to access CSRs and addresses buses besides GPRs. Before writing to command, the debugger should write appropriate data and instructions to the data and progbuf dmi registers to correctly configure the intended debugging operation. The specific count of data and progbuf registers can be configured via the progbufsize and datacount fields of abstractcs.
Once data, and progbuf registers are correctly configured, write to command will mark the beginning of a specific debugging operation, the RC impl has a state machine that handles write of command: when a supported command (val commandWrIsUnsupported = COMMANDWrEn && !commandWrIsAccessRegister)write comes at right time (without a previous command is at play: COMMANDWrEnLegal := (ctrlStateReg === CtrlState(Waiting))) , the state will transfer to CheckGenerate, during this state the command(that is COMMANDWrData) is registered into COMMANDReg and further legality is checked based on COMMANDReg(commandRegBadHaltResume and commandRegIsUnsupported). An Err will make the state transfer back to Waiting, and set corresponding bits in abstractcs:

when (~io.dmactive || ~dmAuthenticated) {
    ABSTRACTCSReg := ABSTRACTCSReset
}.otherwise {
    when (errorBusy){
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrBusy.id.U
    }.elsewhen (errorException) {
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrException.id.U
    }.elsewhen (errorUnsupported) {
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrNotSupported.id.U
    }.elsewhen (errorHaltResume) {
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrHaltResume.id.U
    }.otherwise {
    when (ABSTRACTCSWrEn){
        //hjr cmderr is a RW1C, which means:Used to describe a Read/Write-1-to-Clear register field
        //hjr it means that writing 1 for each bit of this field will actually make it 0;
        ABSTRACTCSReg.cmderr := ABSTRACTCSReg.cmderr & ~(ABSTRACTCSWrData.cmderr);//hjr this is for clearing the error.
    }
    }
}
//hjr 
errorBusy := (ABSTRACTCSWrEnMaybe    && ~ABSTRACTCSWrEnLegal)        ||//hjr try to write ABSTRACTCS
            (autoexecdataWrEnMaybe  && ~ABSTRACTAUTOWrEnLegal)      ||//hjr try to write autoexecdata
            (autoexecprogbufWrEnMaybe && ~ABSTRACTAUTOWrEnLegal)    ||//hjr try to write autoexecprogbuf
            (COMMANDWrEnMaybe       && ~COMMANDWrEnLegal)           ||//hjr try to write COMMAND
            (dmiAbstractDataAccess  && ~dmiAbstractDataAccessLegal) ||//hjr read or write data registers.
            (dmiProgramBufferAccess && ~dmiProgramBufferAccessLegal)//hjr read or write program buffer registers.

If legality check passes and the command is checked not for accessing custom registers(accessRegIsCustom) at CheckGenerate, then the state will go to Exec, a control signal goAbstract is set at the same time:

//------------------------
// Variable ROM STATE MACHINE
// -----------------------
// hjr Waiting->CheckGenerate->Exec
when (ctrlStateReg === CtrlState(Waiting)){
    //hjr if an actual command write is happening, or a data or program buffer register is accessed(the corresponding word is marked in abstractauto),
    // then the command writen process will begin.
    when (wrAccessRegisterCommand || regAccessRegisterCommand) {
    ctrlStateNxt := CtrlState(CheckGenerate)
    }.elsewhen (commandWrIsUnsupported) { // These checks are really on the command type.
    errorUnsupported := true.B
    }.elsewhen (autoexec && commandRegIsUnsupported) {
    errorUnsupported := true.B
    }
//hjr todo have trouble figuring out why this CheckGenerate stata is needed?
}.elsewhen (ctrlStateReg === CtrlState(CheckGenerate)){

    // We use this state to ensure that the COMMAND has been
    // registered by the time that we need to use it, to avoid
    // generating it directly from the COMMANDWrData.
    // This 'commandRegIsUnsupported' is really just checking the
    // AccessRegisterCommand parameters (regno)
    when (commandRegIsUnsupported) {
    errorUnsupported := true.B
    ctrlStateNxt := CtrlState(Waiting)
    }.elsewhen (commandRegBadHaltResume){//hjr todo why there isn't commandWrBadHaltResume version of this?
    errorHaltResume := true.B
    ctrlStateNxt := CtrlState(Waiting)
    }.otherwise {
    when(accessRegIsCustom) {
        ctrlStateNxt := CtrlState(Custom)
    }.otherwise {
        ctrlStateNxt := CtrlState(Exec)
        goAbstract := true.B//hjr true.B marking the begin of command
    }
    }
}

The assertion of goAbstract will drive 2 process:

• Generate instructions for Abstract(0) and Abstract(1)

/*hjr
* In the cycle goReg is true.B(one cycle later than goAbstract is asserted),the instructions in ABSTRACT has been written valid value by following
* logic.
* */
when (goAbstract) {
    if (cfg.nAbstractInstructions == 2) {
    // ABSTRACT(0): Transfer: LW or SW, else NOP
    // ABSTRACT(1): Postexec: NOP       else EBREAK
    abstractGeneratedMem(0) := Mux(accessRegisterCommandReg.transfer,
        //hjr write(1) data-> gpr write(0) gpr->data
        Mux(accessRegisterCommandReg.write, abstractGeneratedI(cfg), abstractGeneratedS(cfg)),
        nop.asUInt()
    )
    abstractGeneratedMem(1) := Mux(accessRegisterCommandReg.postexec,
        nop.asUInt(),
        Instructions.EBREAK.value.U)
    } else {
    // Entry: All regs in GPRs, dscratch1=offset 0x800 in DM
    // ABSTRACT(0): CheckISA: ADDW or NOP (exception here if size=3 and not RV64)
    // ABSTRACT(1):           CSRRW s1,dscratch1,s1 or CSRRW s0,dscratch1,s0
    // ABSTRACT(2): Transfer: LW, SW, LD, SD else NOP
    // ABSTRACT(3):           CSRRW s1,dscratch1,s1 or CSRRW s0,dscratch1,s0
    // ABSTRACT(4): Postexec: NOP else EBREAK
    abstractGeneratedMem(0) := Mux(accessRegisterCommandReg.transfer && accessRegisterCommandReg.size =/= 2.U, isa.asUInt(), nop.asUInt())
    abstractGeneratedMem(1) := abstractGeneratedCSR
    abstractGeneratedMem(2) := Mux(accessRegisterCommandReg.transfer,
        Mux(accessRegisterCommandReg.write, abstractGeneratedI(cfg), abstractGeneratedS(cfg)),
        nop.asUInt()
    )
    abstractGeneratedMem(3) := abstractGeneratedCSR
    abstractGeneratedMem(4) := Mux(accessRegisterCommandReg.postexec,
        nop.asUInt(),
        Instructions.EBREAK.value.U)
    }
}

For now, just concern the case that debug module is at 0x00, see this PR.

• Notifying the hart to begin executing corresponding debugging instruction
  The assertion of goAbstract will assert the goReg (val goReg = Reg(Bool())), this register is used to set the per-hart flag to indicate to hart that debug module has prepared the data and program needed to perform debugging operation, therefore it's time for the hart to go:

/*
* hjr goReg is set by the Variable ROM STATE MACHINE, when an abstract command is written from debugger, goAbstract is asserted,
* goReg will be update to be true.B. This will set the flags:flags(hartSelFuncs.hartSelToHartId(selectedHartReg)).go := goReg;
* The ROM code will be aware of this modification. And the execution flow will jump to WHERETO, inside which there is jal to abstract command.
* Instructions in Abstract is generated by DM in real time according to the cmdtype&transfer&write&postexec bits in the command dmi register.
* It's worth noting that before jump to WHERETO, the ROM code will clear the GO flag:sw zero, GOING(zero), this write will assert the hartGoingWrEn, setting
* goReg being flase.B
* */
val goReg = Reg(Bool())

when (~io.dmactive){
    goReg := false.B
}.otherwise {
    when (goAbstract) {
    goReg := true.B
    }.elsewhen (hartGoingWrEn){//hjr writen by the ROM code---will jump to whereto
    //hjr note that before  the hart executes the WHERETO, it will write 0 to GOING; It's worth noting that
    //hjr it does not write the hartid because there is only one hart among the selected harts will execute the abstract command --the
    //hjr hart selected by hartsel
    assert(hartGoingId === 0.U, "Unexpected 'GOING' hart.")//Chisel3 #540 %x, expected %x", hartGoingId, 0.U)
    goReg := false.B
    }
}
//hjr goReg set by writing an abstract command, cleared by ROM write to GOING.
val flags = WireInit(VecInit(Seq.fill(1 << selectedHartReg.getWidth) {0.U.asTypeOf(new flagBundle())} ))
flags(hartSelFuncs.hartSelToHartId(selectedHartReg)).go := goReg

Note that flags is a vec of wire that is memory mapped to the address space of all harts in system:

val node = dmInner.tlNode
debug.node := tlbus.coupleTo("debug"){ TLFragmenter(tlbus) := _ }
//hjr this tlNode will finally connect to the system bus as a slave
val tlNode = TLRegisterNode(
    address=Seq(cfg.address),
    device=device,
    beatBytes=beatBytes,
    executable=true
)

tlNode.regmap(
    ...,
    FLAGS         -> RegFieldGroup("debug_flags", Some("Memory region used to control hart going/resuming in Debug Mode"),
    if (nComponents == 1) {
        Seq.tabulate(1024) { i => RegField.r(8, flags(0).asUInt(), RegFieldDesc(s"debug_flags_$i", "", volatile=true)) }
    } else {
        flags.zipWithIndex.map{case(x, i) => RegField.r(8, x.asUInt(), RegFieldDesc(s"debug_flags_$i", "", volatile=true))}
    }),
    ...
)

Now let's check out logic of hart side, after the hart halts, the hart is executing in a loop, and waiting for the Debug Module for further operation, and once it sees the change in the FLAGS[hartid].go, the pc will jump to address WHERETO:

//--hjr todo how to guarantee only one hart advances here?
//hjr only the hart selected by hartsel will have the abstract command executed instead of all selected harts.
lbu  s0, FLAGS(s0) // 1 byte flag per hart. Only one hart advances here.Before this loc, s0 has hartid in it.
andi s0, s0, (1 << FLAG_GO) | (1 << FLAG_RESUME)
beqz s0, entry_loop            // Loop until either GO or RESUME is set.
andi s0, s0, (1 << FLAG_GO)
beqz s0, _resume               // If GO is clear at this point, RESUME must be set.

csrr s0, CSR_DSCRATCH          // Restore s0 here
sw zero, GOING(zero)           // When debug module sees this write, the GO flag is reset.
jalr zero, zero, %lo(whereto)          // Rocket-Chip has a specific hack which is that jalr in
                                // Debug Mode will flush the I-Cache. We need that so that the
                                // remainder of the variable instructions will be what Debug Module
                                // intends.

It's worth noting that right after the hart enters debug ROM code, s0 will be stored in CSR_DSCRATCH so that it can be used by the ROM code, like hold value of MHARTID, and before jump to WHERETO, the s0 needs to be restored to the original value:

entry:
    jal zero, _entry
_entry:
    // This fence is required because the execution may have written something
    // into the Abstract Data or Program Buffer registers. hjr
    // hjr todo by whom?--by the debuger req to dmi bus(thru something like jtag dtm).
    // hjr todo what confuses me is whether fence can take effect when previous read&write are from outer debug facility instead of the core.
    fence
    csrw CSR_DSCRATCH, s0  // Save s0 to allow signaling MHARTID---todo hjr for signaling MHARTID???
.......
.......
.......
csrr s0, CSR_DSCRATCH          // Restore s0 here
sw zero, GOING(zero)           // When debug module sees this write, the GO flag is reset.
jalr zero, zero, %lo(whereto)          // Rocket-Chip has a specific hack which is that jalr in
                                // Debug Mode will flush the I-Cache. We need that so that the
                                // remainder of the variable instructions will be what Debug Module
                                // intends.

What stores at WHERETO is a jal x0 abstract instruction:

WHERETO       -> Seq(RegField.r(32, jalAbstract.asUInt, RegFieldDesc("debug_whereto", "Instruction filled in by Debug Module to control hart in Debug Mode", volatile= true))),

Also Note that before the hart pc jumps to WHERETO, and begins to conduct real debugging operations there, hart executes the sw zero, GOING(zero), this acknowledges that hart has already accepted the GO order from debug module, and once the debug module sees this write(indicated by hartGoingWrEn) by hart, it wll deassert the goReg, therefore combinationally deassert the FLAGS[selected].go:

when (~io.dmactive){
    goReg := false.B
}.otherwise {
    when (goAbstract) {
    goReg := true.B
    }.elsewhen (hartGoingWrEn){//hjr written by the ROM code---will jump to whereto
    //hjr note that before  the hart executes the WHERETO, it will write 0 to GOING; It's worth noting that
    //hjr it does not write the hartid because there is only one hart among the selected harts will execute the abstract command --the
    //hjr hart selected by hartsel
    assert(hartGoingId === 0.U, "Unexpected 'GOING' hart.")//Chisel3 #540 %x, expected %x", hartGoingId, 0.U)
    goReg := false.B
    }
}

flags(hartSelFuncs.hartSelToHartId(selectedHartReg)).go := goReg

Note that write to command causes the state transfers to Exec, and the state will be in Exec until an ebreak is met in program buffer or ABSTRACT; According to the spec, The EBREAK instruction is used by debuggers to cause control to be transferred back to a debugging environment. It generates a breakpoint exception and performs no other operation. That is to say, when an ebreak is met, it will generate a breakpoint exception, and traps to M-mode(or delegate to lower priv level according to the configuration of medeleg). Or, the hart may enter debug mode depending on the specific ebreak related fields in dcsr and current hart priv level. However, if a hart has already been in debug mode(indicating by reg_debug), an ebreak will always trap to the beginning of the debug ROM. See code section below:

val cause =
//hjr this is slick, 0x8(user) + 0x0(user) = Environment call from U-mode
//0x8(user) + 0x1(supervisor) = Environment call from S-mode
//0x8(user) + 0x2(machine) = Environment call from M-mode
Mux(insn_call, Causes.user_ecall + Mux(reg_mstatus.prv(0) && reg_mstatus.v, PRV.H: UInt, reg_mstatus.prv),
Mux[UInt](insn_break, Causes.breakpoint, io.cause))
val cause_lsbs = cause(log2Ceil(1 + CSR.busErrorIntCause)-1, 0)
val causeIsDebugInt = cause(xLen-1) && cause_lsbs === CSR.debugIntCause
val causeIsDebugTrigger = !cause(xLen-1) && cause_lsbs === CSR.debugTriggerCause
val causeIsDebugBreak = !cause(xLen-1) && insn_break && Cat(reg_dcsr.ebreakm, reg_dcsr.ebreakh, reg_dcsr.ebreaks, reg_dcsr.ebreaku)(reg_mstatus.prv)

/*
* hjr
* reg_debug below is for handling the debug exception
* */
val trapToDebug = Bool(usingDebug) && (reg_singleStepped || causeIsDebugInt || causeIsDebugTrigger || causeIsDebugBreak || reg_debug)
/*
* hjr I read somewhere that when an exception happens in Debug Mode, the execution flow should go to a specific location
* that is just for handling debug mode exception, maybe it is debugException here.
* */
/*
* hjr--This is very cool.
* 1, an ebreak in the debug mode(reg_debug being high) will jump to the beginning of the ROM code
* 2, If reg_debug is true.B and there is an exception happening, will jump to the debug exception handler:debugException
* 3, When reg_debug is false.B, this means there is a haltreq(or ebreak for DebugMode, or debugTrigger), just jump to debugEntry
* */
val debugTVec = Mux(reg_debug, Mux(insn_break, debugEntry.U, debugException.U), debugEntry.U)
val tvec = Mux(trapToDebug, debugTVec, Mux(trapToNmi, nmiTVec, notDebugTVec))
io.evec := tvec

The code above is inside the CSR.scala, for handling the debug related interrupt and exception. We can see that the execution flow(io.evec) will jump to debugEntry when an ebreak is met in debug mode(indicated by the reg_debug), because trapToDebugis always asserted in debug mode: val trapToDebug = Bool(usingDebug) && (reg_singleStepped || causeIsDebugInt || causeIsDebugTrigger || causeIsDebugBreak || reg_debug), note the reg_debug. This also indicates that if we met some unexpected scenario in debug mode, the debug mode will always be re-trapped, instead of trapping into other priv levels.
After the program buffer executes ebreak, that means another command can be accepted by debug module, therefore the state should be transferred back to Waiting from Exec, this is guaranteed by the following code:

...
    }.elsewhen (ctrlStateReg === CtrlState(Exec)) {

    // We can't just look at 'hartHalted' here, because
    // hartHaltedWrEn is overloaded to mean 'got an ebreak'
    // which may have happened when we were already halted.
    when(goReg === false.B && hartHaltedWrEn && (hartSelFuncs.hartIdToHartSel(hartHaltedId) === selectedHartReg)){
    //hjr an ebreak inside debug mode is met, just jump to ROM start, and set the state machine to be Waiting
    ctrlStateNxt := CtrlState(Waiting)
    }
    when(hartExceptionWrEn) {
    //hjr todo why hartExceptionId === 0.U??
    assert(hartExceptionId === 0.U, "Unexpected 'EXCEPTION' hart")//Chisel3 #540, %x, expected %x", hartExceptionId, 0.U)
        ctrlStateNxt := CtrlState(Waiting)
    errorException := true.B
    }
}

Note code above that in the state of Exec, when goReg === false.B && hartHaltedWrEn && (hartSelFuncs.hartIdToHartSel(hartHaltedId) === selectedHartReg), this normally means an ebreak is met in program buffer or in ABSTRACT:goReg === false.B indicates that the hart has acknowledged the "GO" order from DM, then hart will jump to WEHRETO, beginning executing the debugging operation represented by the instructions generated in ABSTRACT and the program buffer. hartHaltedWrEn is asserted by writes to HALTED, this happens when the hart is re-entered into the Debug Rom(remember that right after entering the debug rom, hart will write to HALTED its hartid acknowledging to debug module), normally an ebreak in program buffer or ABSTRACT will fulfil this goReg === false.B && hartHaltedWrEn. Consequently hartHaltedWrEn is overloaded to mean 'got an ebreak' in this scenario;
Also note that when an exception happens during the execution of program buffer, then the execution flow will jump to a specific location inside the debug module:

val debugException = p(DebugModuleKey).map(_.debugException).getOrElse(BigInt(0x808))
val debugTVec = Mux(reg_debug, Mux(insn_break, debugEntry.U, debugException.U), debugEntry.U)
val trapToDebug = Bool(usingDebug) && (reg_singleStepped || causeIsDebugInt || causeIsDebugTrigger || causeIsDebugBreak || reg_debug)

Note the code above that the reg_debug & !insn_break just means an exception happens in debug mode. If an exception happens in debug mode, reg_debug will stop that exception from handling as normal exception, instead will always trapToDebug and jump to address of debugException.
Summing up, the debug module can use command scheme depicted above to execute instructions generated by debug module to conduct specific high level debugging intents. I previously have confusion on what marks begin and end of debug mode, now it's clear: the reg_debug, when the DM first fires a debug interrupt due to request from debugger, the reg_debug will be asserted, and the execution flow will jumps to debug rom (io.evec := tvec):

val exception = insn_call || insn_break || io.exception
when (exception) {//hjr exception and interrupt are regarded as "exception"
  when (trapToDebug) {
    //hjr when trapToDebug and reg_debug are both asserted, this happens at the moment when debug ROM code has an exception or meets an ebreak
    //hjr under this situation, just jump to the debug exception handler(0x808) or debug entry, do nothing here, debug mode register is not destoryed.
    when (!reg_debug) {
      reg_mstatus.v := false//hjr todo what's this for? H extension stuff.
      reg_debug := true
      reg_dpc := epc
      reg_dcsr.cause := Mux(reg_singleStepped, 4, Mux(causeIsDebugInt, 3, Mux[UInt](causeIsDebugTrigger, 2, 1)))
      reg_dcsr.prv := trimPrivilege(reg_mstatus.prv)
      reg_dcsr.v := reg_mstatus.v
      new_prv := PRV.M//hjr debug mode and m-mode are basically the same.
    }
  }
  .....
}

ebreak in program buffer will transfer the execution flow back to the Debug ROM again, and the reg_debug is still asserted, means that the ebreak does not end debug mode. Instead, the debug mode is ended via explicit request from debugger by asserting the resume request in dmcontrol: resumereq(sending resume request to all selected harts);
Once an ebreak is met when executing a command, the hart pc will jump to ROM code, and then waits in loop for further order from the debug module, the debugger can then execute another command by writing to data, progbufand command. After finish all intended operations, the debugger can ask the hart to resume execution via writing to dmcontrol.resumereq, TLDebugModuleOuter handles dmcontrol related logic and feed the resume request signal into TLDebugModuleInner (val resumereq = io.innerCtrl.fire() && io.innerCtrl.bits.resumereq), this will set corresponding bits in resumeReqRegs(indicating that a resumeReq for a specific hart is ongoing) for selected harts(the selected harts are represented by bits in hamaskWrSel). Asserted bits in resumeReqRegs will combinationally set the corresponding bits in flags for selected harts. Hart now is executing in the ROM loop, and will be aware this flags value change, and therefore will jump to _resume, code inside the _resume section will write its hartid to RESUMING, debug module will be aware of this write(via hartResumingWrEn and hartResumingId)and will clear the related bits in haltedBitRegs(meaning the select hart will snap out of halt mode), also the dm will clear the corresponding bits in resumeReqRegs(meaning the resume request has been acked by hart.) The code is as follows:

when (~io.dmactive || ~dmAuthenticated) {
    haltedBitRegs := 0.U
    resumeReqRegs := 0.U
    }.otherwise {

    resumeReqRegs := resumeReqRegs & ~(hartIsInResetSync.asUInt)//hjr reset signal will mask the resume req.

    val hartHaltedIdIndex   = UIntToOH(hartSelFuncs.hartIdToHartSel(hartHaltedId))
    val hartResumingIdIndex = UIntToOH(hartSelFuncs.hartIdToHartSel(hartResumingId))
    val hartselIndex        = UIntToOH(io.innerCtrl.bits.hartsel)
    when (hartHaltedWrEn) {
        //hjr haltedBitRegs is set by ROM code writing to HALTED.
        /*
        * hjr A halt request is initiated by the debugger writing to dmcontro.haltreq, this will fire an interrupt to hartsel AND hartse harts,
        * causing the hart begin executing the ROM code serviced from debug module. The ROM code will sw   s0, HALTED(zero) which will cause haltedBitRegs
        * being set.
        *
        * */
        haltedBitRegs := (haltedBitRegs | hartHaltedIdIndex) & ~(hartIsInResetSync.asUInt)
    }.elsewhen (hartResumingWrEn) {
        haltedBitRegs := (haltedBitRegs & ~(hartResumingIdIndex)) & ~(hartIsInResetSync.asUInt)
    }.otherwise {
        /*
        * hjr a reset signal will destroy the halt state
        * The spec says:
        * Otherwise, if the hart was initially running it will execute normally (running state)
        * and if the hart was initially halted it should now be running but may be halted.
        * */
        haltedBitRegs := haltedBitRegs & ~(hartIsInResetSync.asUInt)
    }
    /*
    * hjr a resume request is initiated by the debugger writing to the dmcontrol.resumereq, this will set corresponding bits in resumeReqRegs, and combinationally
    * set the flag.resume of selected harts. The ROM code will see this value andi s0, s0, (1 << FLAG_GO) | (1 << FLAG_RESUME), and jump to the resume:
    * _resume:
    *  csrr s0, CSR_MHARTID
    *  sw   s0, RESUMING(zero) // When Debug Module sees this write, the RESUME flag is reset.
    *  csrr s0, CSR_DSCRATCH   // Restore s0
    *  dret
    *  Inside the resume section, it will restore the s0, and set corresponding RESUMING indicator, meaning that the hart is handling the resume request.Therefore
    * the resumeReqRegs needs to be cleared.
    * */
    //hjr hartResumingWrEn is set by the ROM code, sw s0, RESUMING(zero),which means the hart acks this resume request initiated by debugger
    //so, the bit in resumeReqRegs will be cleared.
    // When Debug Module sees this write, the RESUME flag is reset.
    when (hartResumingWrEn) {
        resumeReqRegs := (resumeReqRegs & ~(hartResumingIdIndex)) & ~(hartIsInResetSync.asUInt)
    }
    //hjr setting by the debugger explicitly asking the selected hart to resume.
    when (resumereq) {
        resumeReqRegs := (resumeReqRegs | hamaskWrSel.asUInt) & ~(hartIsInResetSync.asUInt)
    }

    }
    /*
    * bits inisde the resumeReqRegs is asserted by debugger writing to the dmcontrl.resumereq, cleared by ROM code writing to RESUMING.
    * resumeAcks is used for indicating whether a hart has acknowledged the resume request from debugger(by writing its hartid in RESUMING),
    * it's used to caculate the `anyresumeack` and `allresumeack` of `dmstatus`.
    * Since hartResumingWrEn will have the corresponding bit in resumeReqRegs deasserted, therefore normally resumeAcks can be
    * indicated as ~resumeReqRegs. However, it's worth noting that when resumereq is asserted, according to the code above, the bits of 
    * resumeReqRegs that corresponds to the selected harts will be asserted next cycle. Therefore these corresponding bits in resumeAcks should * not be set under this circumstance.
    * */
    when (resumereq) {//hjr 1 cycle before the resumeReqRegs is set.
    resumeAcks := (~resumeReqRegs & ~(hamaskWrSel.asUInt))
    }.otherwise {
    resumeAcks := ~resumeReqRegs
    }

After sw s0, RESUMING(zero) , the hart will continue executing; Note that when entering the debug mode, s0 register is saved so that it can be used as debugging purpose, since we are leaving the debug mode now, s0 shall be restored:csrr s0, CSR_DSCRATCH(the value is saved in the dscratch).
After doing all above, the hart will execute one last instruction in debug mode: dret, like other xrets , dret is an instruction that only has meaning while in Debug Mode. Its recommended encoding is 0x7b200073. When dret is executed, pc is restored from dpc and normal execution resumes at the privilege level set by prv in dcsr. Also, dret will deassert the reg_debug, marking the real end of debug mode:

when (insn_ret) {
    ...
    elsewhen (Bool(usingDebug) && io.rw.addr(10) && io.rw.addr(7)) {//hjr dret
        ret_prv := reg_dcsr.prv
        reg_mstatus.v := usingHypervisor && reg_dcsr.v && reg_dcsr.prv <= PRV.S
        reg_debug := false//hjr  from haltreq to dret
        io.evec := readEPC(reg_dpc)
    }
    new_prv := ret_prv
}

...
val new_prv = Wire(init = reg_mstatus.prv)
reg_mstatus.prv := legalizePrivilege(new_prv)

Except for the general debugging process depicted above(entering debug mode via haltreq, writing command to conduct specific debugging operation, then exiting the debug mode via resumereq), there are extra details need to clarify in terms of documentation completeness, I will clarify them in detail as follows:

• How selected harts are represented
  A large portion of debugging operations are applied to selected harts, refer to setcion3.3 of the riscv debug spec for detailed info on how to select a single or multiple harts. Note that even though the debugging operations like halt, resume and dmreset are applied to all selected harts, execution of Abstract Commands ignores this mechanism and only applies to the hart selected by hartsel.
  The logic for selecting harts are handled in the Outer, so that specific harts can be selected even though the hart is not running(the TLDebugModuleOuterAsync and TLDebugModuleInnerAsync work under different clock domain). The outer will send the hamask(represent the harts selected by hasel) and hartsel to inner, and selectedHartReg in the inner indicates the hart selected by hartsel, and hamaskFull in the inner is a vector of all selected harts including hartsel, whether or not supportHartArray is true. The related code is straightforward as follows:

val selectedHartReg = Reg(UInt(p(MaxHartIdBits).W))
// hamaskFull is a vector of all selected harts including hartsel, whether or not supportHartArray is true
val hamaskFull = WireInit(VecInit(Seq.fill(nComponents) {false.B} ))

if (nComponents > 1) {
    when (~io.dmactive) {
    selectedHartReg := 0.U
    }.elsewhen (io.innerCtrl.fire()){
    selectedHartReg := io.innerCtrl.bits.hartsel
    }
}

if (supportHartArray) {
    val hamaskZero = WireInit(VecInit(Seq.fill(nComponents) {false.B} ))
    val hamaskReg = Reg(Vec(nComponents, Bool()))
    when (~io.dmactive || ~dmAuthenticated) {
    hamaskReg := hamaskZero
    }.elsewhen (io.innerCtrl.fire()){
    hamaskReg := Mux(io.innerCtrl.bits.hasel, io.innerCtrl.bits.hamask, hamaskZero)
    }
    hamaskFull := hamaskReg
}

when (selectedHartReg < nComponents.U) {
    hamaskFull(selectedHartReg) := true.B
}

io.innerCtrl.ready := true.B
//hjr todo what's the difference between hamaskWrSel and hamaskFull
// Construct a Vec from io.innerCtrl fields indicating whether each hart is being selected in this write
// A hart may be selected by hartsel field or by hart array
val hamaskWrSel = WireInit(VecInit(Seq.fill(nComponents) {false.B} ))
for (component <- 0 until nComponents ) {
    hamaskWrSel(component) := ((io.innerCtrl.bits.hartsel === component.U) ||
    (if (supportHartArray) io.innerCtrl.bits.hasel && io.innerCtrl.bits.hamask(component) else false.B))
}

The hamask is constructed in outer by writing to hasel(enable the hart array mask functionality), hawindowsel(select a specific window to set up), and hawindow(set up a sepcific window, write 1 to each bit means that hart is selected)

• Halt On Reset Logic
  The functionality of halt on reset is configured by setresethaltreq and clrresethaltreq fields in dmcontrol, refer to the debug spec for detailed depiction. In short words, asserted bit in setresethaltreq indicates that selected harts will enter halting mode after snaping out of reset. The halt-on-reset configuration can only be cleared by asserting clrresethaltreq(Spec says if both setresethaltreq and clrresethaltreq are 1, then clrresethaltreq is executed). The dmcontrol logic is handled at the TLDebugModuleOuter, there is a hrmaskReg in Outer that indicates whether a specific hart should hatl on reset, the logic is pretty staightforward:

//--------------------------------------------------------------
// Halt-on-reset
//  hrmaskReg is current set of harts that should halt-on-reset
//    Reset state (dmactive=0) is all zeroes
//    Bits are set by writing 1 to DMCONTROL.setresethaltreq
//    Bits are cleared by writing 1 to DMCONTROL.clrresethaltreq
//    Spec says if both are 1, then clrresethaltreq is executed
//--------------------------------------------------------------
val hrmask    = Wire(Vec(nComponents, Bool()))
val hrmaskNxt = Wire(Vec(nComponents, Bool()))
val hrmaskReg = RegNext(next=hrmaskNxt, init=0.U.asTypeOf(hrmaskNxt)).suggestName("hrmaskReg")
/*
* hjr hrmaskNxt is input of the hrmaskReg and also the output of hrmaskReg so that hrmask will have value set at very beginning
* hjr instead of 1 cycle later.
* todo needs confirm
* */
hrmaskNxt := hrmaskReg
for (component <- 0 until nComponents) {
    when (~dmactive || ~dmAuthenticated) {
    hrmaskNxt(component) := false.B
    }.elsewhen (clrresethaltreqWrEn && DMCONTROLWrData.clrresethaltreq && hartSelected(component)) {
    hrmaskNxt(component) := false.B
    }.elsewhen (setresethaltreqWrEn && DMCONTROLWrData.setresethaltreq && hartSelected(component)) {
    hrmaskNxt(component) := true.B
    }
}
hrmask := hrmaskNxt

The hrmask will be sent to the inner part for further handling via the innerCtrl bundle along with other signals that's needed by the inner part:

// These registers ensure that requests to dmInner are not lost if inner clock isn't running or requests occur too close together.
// If the innerCtrl async queue is not ready, the notification will be posted and held until ready is received.
// Additional notifications that occur while one is already waiting update the pending data so that the last value written is sent.
// Volatile events resumereq and ackhavereset are registered when they occur and remain pending until ready is received.
val innerCtrlValid = Wire(Bool())
val innerCtrlValidReg = RegInit(false.B).suggestName("innerCtrlValidReg")
val innerCtrlResumeReqReg = RegInit(false.B).suggestName("innerCtrlResumeReqReg")
val innerCtrlAckHaveResetReg = RegInit(false.B).suggestName("innerCtrlAckHaveResetReg")

innerCtrlValid := hartselloWrEn | resumereqWrEn | ackhaveresetWrEn | setresethaltreqWrEn | clrresethaltreqWrEn | haselWrEn |
    (HAWINDOWWrEn & supportHartArray.B)

innerCtrlValidReg        := io.innerCtrl.valid & ~io.innerCtrl.ready             // Hold innerctrl request until the async queue accepts it
innerCtrlResumeReqReg    := io.innerCtrl.bits.resumereq & ~io.innerCtrl.ready    // Hold resumereq until accepted
innerCtrlAckHaveResetReg := io.innerCtrl.bits.ackhavereset & ~io.innerCtrl.ready // Hold ackhavereset until accepted

io.innerCtrl.valid             := innerCtrlValid | innerCtrlValidReg
io.innerCtrl.bits.hartsel      := Mux(hartselloWrEn, DMCONTROLWrData.hartsello, DMCONTROLReg.hartsello)
io.innerCtrl.bits.resumereq    := (resumereqWrEn & DMCONTROLWrData.resumereq) | innerCtrlResumeReqReg
io.innerCtrl.bits.ackhavereset := (ackhaveresetWrEn & DMCONTROLWrData.ackhavereset) | innerCtrlAckHaveResetReg
io.innerCtrl.bits.hrmask       := hrmask
if (supportHartArray) {
    io.innerCtrl.bits.hasel      := Mux(haselWrEn, DMCONTROLWrData.hasel, DMCONTROLReg.hasel)
    io.innerCtrl.bits.hamask     := hamask
}

The inner part will use hrmask to decide whether a reset will cause the specific hart halting: if specific bit in hrmask is set, and that hart is undergoing a reset(indicated by the hartIsInResetSync) , corresponding bit in hrDebugIntReg will be set, indicating that the hart should halt immdeidately on the next deassertion of its reset. I still have trouble figuring out the sync and async reset stuff:

//-------------------------------------
// Halt-on-reset logic
//  hrmask is set in dmOuter and passed in
//  Debug interrupt is generated when a reset occurs whose corresponding hrmask bit is set
//  Debug interrupt is maintained until the hart enters halted state
//-------------------------------------
val hrReset    = WireInit(VecInit(Seq.fill(nComponents) { false.B } ))
val hrDebugInt = Wire(Vec(nComponents, Bool()))
val hrmaskReg  = RegInit(hrReset)
val hartIsInResetSync = Wire(Vec(nComponents, Bool()))

for (component <- 0 until nComponents) {
    hartIsInResetSync(component) := AsyncResetSynchronizerShiftReg(io.hartIsInReset(component), 3, Some(s"debug_hartReset_$component"))
}

when (~io.dmactive || ~dmAuthenticated) {
    hrmaskReg := hrReset
}.elsewhen (io.innerCtrl.fire()){
    hrmaskReg := io.innerCtrl.bits.hrmask
}

withReset(reset.asAsyncReset) {          // ensure interrupt requests are negated at first clock edge
    val hrDebugIntReg = RegInit(VecInit(Seq.fill(nComponents) { false.B } ))
    when (~io.dmactive || ~dmAuthenticated) {
    hrDebugIntReg := hrReset
    }.otherwise {
    hrDebugIntReg := hrmaskReg &
    //hjr hart on reset
        (hartIsInResetSync |               // set debugInt during reset
        (hrDebugIntReg & ~(haltedBitRegs.asBools)))  // maintain until core halts
    }
    hrDebugInt := hrDebugIntReg
}

Note that individual signal in hrDebugIntReg will keep asserting until the halt request (initiated by reset) has been acked by hart:(hrDebugIntReg & ~(haltedBitRegs.asBools)); hrDebugInt is used to interrupt the hart, make it trap to debug mode:io.hgDebugInt := hgDebugInt | hrDebugInt
In terms of general reset related logic, refer to 3.2 of debug spec for detailed info; Specifically, the debug module have 2 methods to reset harts:
ndmreset resets all the harts in the hardware platform, as well as all other parts of the hardware platform except for the Debug Modules, Debug Transport Modules, and Debug Module Interface:

class TestHarness()(implicit p: Parameters) extends Module {
  val io = IO(new Bundle {
    val success = Output(Bool())
  })

  val ldut = LazyModule(new ExampleRocketSystem)
  println("end of the first stage")
  val dut = Module(ldut.module)
  // Allow the debug ndreset to reset the dut, but not until the initial reset has completed
  //hjr see https://github.com/chipsalliance/rocket-chip/commit/63f6cd1180fda066b0f22bad8034bcac8d58cdc2
  dut.reset := (reset.asBool | dut.debug.map { debug => AsyncResetReg(debug.ndreset) }.getOrElse(false.B)).asBool

  dut.dontTouchPorts()
  dut.tieOffInterrupts()
  SimAXIMem.connectMem(ldut)
  SimAXIMem.connectMMIO(ldut)
  ldut.l2_frontend_bus_axi4.foreach(_.tieoff)
  Debug.connectDebug(dut.debug, dut.resetctrl, dut.psd, clock, reset.asBool, io.success)
}

Also, the debug module can reset selected harts specfically by asserting the hartreset in dmcontrl, this will assert the corresponding bit in io.hartResetReq of TLDebugModuleOuter for selected harts, this signal will propagate through the debug module into the selected harts.

//In TLDebugModuleOuter:
if (cfg.hasHartResets) {
  val hartResetNxt = Wire(Vec(nComponents, Bool()))
  val hartResetReg = RegNext(next=hartResetNxt, init=0.U.asTypeOf(hartResetNxt))

  for (component <- 0 until nComponents) {
    hartResetNxt(component) := DMCONTROLReg.hartreset & hartSelected(component)
    io.hartResetReq.get(component) := hartResetReg(component)
  }
}
//In TLDebugModuleOuterAync: 
io.hartResetReq.foreach { x => dmOuter.module.io.hartResetReq.foreach {y => x := y}}//hjr todo no sync???
//In TLDebugModule
io.hartResetReq.foreach { x => dmOuter.module.io.hartResetReq.foreach {y => x := y}}
//In HasPeripheryDebugModuleImp
val resetctrl = outer.debugOpt.map { outerdebug =>
  outerdebug.module.io.tl_reset := outer.debugTLDomainOpt.get.in.head._1.reset
  outerdebug.module.io.tl_clock := outer.debugTLDomainOpt.get.in.head._1.clock
  val resetctrl = IO(new ResetCtrlIO(outerdebug.dmOuter.dmOuter.intnode.edges.out.size))
  outerdebug.module.io.hartIsInReset := resetctrl.hartIsInReset
  resetctrl.hartResetReq.foreach { rcio => outerdebug.module.io.hartResetReq.foreach { rcdm => rcio := rcdm }}
  resetctrl
}

Note code above that hartResetReq is for reset request initiated by debug module(the outer part specifically, dmcontrl is handled in the outer part), while hartIsInReset comes from the platform into the debug module indicating that a hart's reset signal has been asserted:

resetctrlOpt.map { rcio => rcio.hartIsInReset.map { _ := r }}

Loc above indicates that individual bits inside hartIsInReset that corresponds to harts in the system are all drived by the system level reset signal r instead of individual hart reset. Also note that hartIsInReset actually goes into the inner part: hartIsInResetSync(component) := AsyncResetSynchronizerShiftReg(io.hartIsInReset(component), 3, Some(s"debug_hartReset_$component")), I thought there is no need to add cross domain facility here because the inner part of the DTM and the platform may work under same clock domain, I need more knowledge of domain crossing stuff to make this clear.
hartIsInReset will cause corresponding bit in haveResetBitRegs (inner) being asserted. haveResetBitRegs is used by debug module to notify the reset staus of controlled harts:

DMSTATUSRdData.anyhavereset := (haveResetBitRegs.asBools &  hamaskFull).reduce(_ | _)
DMSTATUSRdData.allhavereset := (haveResetBitRegs.asBools | ~hamaskFull).reduce(_ & _)

The debugger can write to ackhavereset of dmcontrol, indicating the reset of selected harts has been acked by debugger, therefore the corresponding bits in haveResetBitRegs should be cleared:

when(~io.dmactive || ~dmAuthenticated) {
  haveResetBitRegs := 0.U
}.otherwise {
  when (io.innerCtrl.fire() && io.innerCtrl.bits.ackhavereset) {//hjr ackhavereset is written by debugger, clear havereset bits of selected hart.
    //hjr if the ackhavereset is written by the outter, the selected harts's haveReset bits should be cleared.
    haveResetBitRegs := (haveResetBitRegs & (~(hamaskWrSel.asUInt))) | hartIsInResetSync.asUInt
  }.otherwise {
    haveResetBitRegs := haveResetBitRegs | hartIsInResetSync.asUInt 
  }
}

One thing that confuses me is that hartResetReq does not actually drive the reset signal of selected harts. Maybe I need to confirm this;

• Halt Group Implementation(DMCS2)
  The Halt Group utility is useful when you want to halt a group of related harts simultaneously. In RISCV debug spec, resume group is also defined, only the halt group is supported however in RC impl. Refer to the section 3.6 for further introduction on halt&resume group.
  In terms of halt group, each hart has an group number(stored inside hgParticipateHart), when any hart inside the same halt group halts(indicate by bits in haltedBitRegs which means the halting request for that hart has been acked by the hart itself), or any external triggers(the group number of each trigger is stored in hgParticipateTrig) which belongs to that group is firing(val trigInReq = ResetSynchronizerShiftReg(in=extTriggerInReq, sync=3, name=Some("dm_extTriggerInReqSync"))), the hart will halt and trap to debug mode, the cause in dcsr will be marked as 6(6 means halt group--However, the RC impl only reports 3 in this scenario);
  According to the spec, external triggers are abstract concepts that can signal the DM and/or receive signals from the DM. External triggers could be used to implement near simultaneous halting/resuming of all cores in a hardware platform, when not all cores are RISC-V cores. The corresponding signal in terms of external triggers is as follows, they are directly sent to(from) inner from(to) platforms instead of outer:

class DebugExtTriggerIn (val nExtTriggers: Int) extends Bundle {
    val req = Input(UInt(nExtTriggers.W))
    val ack = Output(UInt(nExtTriggers.W))
}
class DebugExtTriggerOut (val nExtTriggers: Int) extends Bundle {
  val req = Output(UInt(nExtTriggers.W))
  val ack = Input(UInt(nExtTriggers.W))
}

class DebugExtTriggerIO () (implicit val p: Parameters) extends ParameterizedBundle()(p) {
    val out = new DebugExtTriggerOut(p(DebugModuleKey).get.nExtTriggers)
    val in  = new DebugExtTriggerIn (p(DebugModuleKey).get.nExtTriggers)
}

The logic needed to handle halt groups can be divided into 2 parts: configuration of group and other fields in dmcs2 and firing detection of a specific group.
The group configuration of a hart is specified by writing to group field group number(starting from 0, and increase continuously, max value is impl dependent) the hart(or the external trigger) should be placed into, and assert the hgwrite to indicate that this is a group configuration, instead of merely a group obtain(read) from debugger(The value written to group field is ignored unless hgwrite is also written 1). Field hgselect affects whether this is for configuring the harts(hgselect being 0) or external trigger(hgselect being 1); If the configuration is for external trigger(hgselect being 1), only group of external trigger indicated in dmexttrigger will be configured, while group configuration for harts(hgselect being 0) will affect the group of all selected harts(not just hartsel). It's worth noting that when the debugger just wants to obtain group instead of setting it(that is, when hgwrite is deasserted), the group obtained by read group field will be group of the hartsel(DMCS2RdData.haltgroup := hgParticipateHart(selectedHartReg)) when hgselect being 0 and group of external trigger(DMCS2RdData.haltgroup := hgParticipateTrig(hgExtTrigger)) dmexttrigger when hgselect being 1; The corresponding code is as follows:

//----DMCS2 (Halt Groups)

val DMCS2RdData    = WireInit(0.U.asTypeOf(new DMCS2Fields()))
val DMCS2WrData    = WireInit(0.U.asTypeOf(new DMCS2Fields()))
val hgselectWrEn   = WireInit(false.B)
val hgwriteWrEn    = WireInit(false.B)
val haltgroupWrEn  = WireInit(false.B)
val exttriggerWrEn = WireInit(false.B)
val hgDebugInt     = WireInit(VecInit(Seq.fill(nComponents) {false.B} ))
//hjr todo why?-->async reset ensures triggers don't falsely fire during startup
if (nHaltGroups > 0) withReset (reset.asAsyncReset) {     // async reset ensures triggers don't falsely fire during startup
    val hgBits = log2Up(nHaltGroups)
    // hgParticipate: Each entry indicates which hg that entity belongs to (1 to nHartGroups). 0 means no hg assigned.
    val hgParticipateHart = RegInit(VecInit(Seq.fill(nComponents)(0.U(hgBits.W))))
    // hjr each entry indicates which hg that trigger belongs to (1 to nHartGroups). 0 means no hg assigned.
    val hgParticipateTrig = if (nExtTriggers > 0) RegInit(VecInit(Seq.fill(nExtTriggers)(0.U(hgBits.W)))) else Nil

    for (component <- 0 until nComponents) {
    when (~io.dmactive || ~dmAuthenticated) {
        hgParticipateHart(component) := 0.U
    }.otherwise {
        //hjr hgwrite means that a modification attempt to the belonging group of harts or triggers(decided by the assertion of hgselect)
        //hjr if hgselect is 0, the group of extrigger(dmexttrigger) will be changed to haltgroup
        //hjr if hgselect is 1, the group of selected harts(hamaskFull(component) means hartsel and hart array) will be changed to haltgroup
        when (haltgroupWrEn & DMCS2WrData.hgwrite & ~DMCS2WrData.hgselect &
            hamaskFull(component) & (DMCS2WrData.haltgroup <= nHaltGroups.U)) {
        hgParticipateHart(component) := DMCS2WrData.haltgroup
        }
    }
    }
    //hjr this complies with the spec, read of the group field will return the group of the hartsel.
    //hjr revised comment: this does not actually comply with the spec, the sepc just says:
    //hjr When hgselect is 0, contains the group of the hart specified by hartsel.
    DMCS2RdData.haltgroup := hgParticipateHart(selectedHartReg)

    if (nExtTriggers > 0) {
    val hgSelect = Reg(Bool())

    when (~io.dmactive || ~dmAuthenticated) {
        hgSelect := false.B
    }.otherwise {
        when (hgselectWrEn) {
            hgSelect := DMCS2WrData.hgselect
        }
    }

    for (trigger <- 0 until nExtTriggers) {
        when (~io.dmactive || ~dmAuthenticated) {
        hgParticipateTrig(trigger) := 0.U
        }.otherwise {
        when (haltgroupWrEn & DMCS2WrData.hgwrite & DMCS2WrData.hgselect &
            (DMCS2WrData.exttrigger === trigger.U) & (DMCS2WrData.haltgroup <= nHaltGroups.U)) {
            hgParticipateTrig(trigger) := DMCS2WrData.haltgroup
        }
        }
    }

    DMCS2RdData.hgselect := hgSelect
    when (hgSelect) {
        //hjr If there is only 1 ext trigger, then the exttrigger field is fixed at 0
        //hjr so, this means only 1 external trigger in the system?
        DMCS2RdData.haltgroup := hgParticipateTrig(0)
    }

    // If there is only 1 ext trigger, then the exttrigger field is fixed at 0
    // Otherwise, instantiate a register with only the number of bits required

    if (nExtTriggers > 1) {
        val trigBits = log2Up(nExtTriggers-1)
        val hgExtTrigger = Reg(UInt(trigBits.W))
        when (~io.dmactive || ~dmAuthenticated) {
        hgExtTrigger := 0.U
        }.otherwise {
        when (exttriggerWrEn & (DMCS2WrData.exttrigger < nExtTriggers.U)) {
            hgExtTrigger := DMCS2WrData.exttrigger
        }
        }

        DMCS2RdData.exttrigger := hgExtTrigger
        when (hgSelect) {
        //hjr the haltgroup when hgSelect is asserted is the haltgroup of the exttrigger specified  in exttrigger field.
        DMCS2RdData.haltgroup := hgParticipateTrig(hgExtTrigger)
        }
    }
    }

In terms of group firing detetcion, we need to detect both the halting of harts (hgHartFiring) and the firing of external triggers(hgTrigFiring) in one specific group, specific bit in signal hgFired indicates that a specific halt group is firing, meaning that a hart in that group has(the way we say a hart is halted is that the halting request from debug module has been acked by the hart) halted(hgHartFiring(hg) := hartHaltedWrEn & ~haltedBitRegs(hartHaltedId) & (hgParticipateHart(hartSelFuncs.hartIdToHartSel(hartHaltedId)) === hg.U)) or an external trigger in the same group is firing(hgTrigFiring(hg) := (trigInReq & ~RegNext(trigInReq) & hgParticipateTrig.map(_ === hg.U)).reduce(_ | _)):

    // Halt group state machine
    //  IDLE:  Go to FIRED when any hart in this hg writes to HALTED while its HaltedBitRegs=0
    //                     or when any trigin assigned to this hg occurs
    //  FIRED: Back to IDLE when all harts in this hg have set their haltedBitRegs
    //                     and all trig out in this hg have been acknowledged

    val hgFired          = RegInit (VecInit(Seq.fill(nHaltGroups+1) {false.B} ))//hjr +1 for group 0?
    val hgHartFiring     = WireInit(VecInit(Seq.fill(nHaltGroups+1) {false.B} ))     // which hg's are firing due to hart halting
    val hgTrigFiring     = WireInit(VecInit(Seq.fill(nHaltGroups+1) {false.B} ))     // which hg's are firing due to trig in
    val hgHartsAllHalted = WireInit(VecInit(Seq.fill(nHaltGroups+1) {false.B} ))     // in which hg's have all harts halted
    val hgTrigsAllAcked  = WireInit(VecInit(Seq.fill(nHaltGroups+1) { true.B} ))     // in which hg's have all trigouts been acked

    io.extTrigger.foreach { extTrigger =>
    val extTriggerInReq = Wire(Vec(nExtTriggers, Bool()))
    val extTriggerOutAck = Wire(Vec(nExtTriggers, Bool()))
    extTriggerInReq := extTrigger.in.req.asBools
    extTriggerOutAck := extTrigger.out.ack.asBools//hjr coming into the debug module from outter trigger.
    val trigInReq  = ResetSynchronizerShiftReg(in=extTriggerInReq,  sync=3, name=Some("dm_extTriggerInReqSync"))
    val trigOutAck = ResetSynchronizerShiftReg(in=extTriggerOutAck, sync=3, name=Some("dm_extTriggerOutAckSync"))
    for (hg <- 1 to nHaltGroups) {
        //hjr todo why ~RegNext(trigInReq)=>only the first cycle trigInReq is high.
        //hjr for each haltgroup, check whether an exttrigger that belongs to that group(hgParticipateTrig.map(_ === hg.U)) is firing
        hgTrigFiring(hg) := (trigInReq & ~RegNext(trigInReq) & hgParticipateTrig.map(_ === hg.U)).reduce(_ | _)
        //hjr for each haltgroup, check whether all firing exttriggers inside this group has been acked.
        hgTrigsAllAcked(hg) := (trigOutAck | hgParticipateTrig.map(_ =/= hg.U)).reduce(_ & _)
    }
    extTrigger.in.ack := trigInReq.asUInt()
    }

    for (hg <- 1 to nHaltGroups) {
    //hjr for each haltgroup, check whether any hart that belongs to this group has set the HALTED bit(before the ~haltedBitRegs(hartHaltedId))
    hgHartFiring(hg) := hartHaltedWrEn & ~haltedBitRegs(hartHaltedId) & (hgParticipateHart(hartSelFuncs.hartIdToHartSel(hartHaltedId)) === hg.U)
    // hjr for each haltgroup, check any hart that belongs to this group has set the bit in haltedBitRegs
    // (which means that this HALTED operation has been acked by dm)
    hgHartsAllHalted(hg) := (haltedBitRegs.asBools | hgParticipateHart.map(_ =/= hg.U)).reduce(_ & _)

    when (~io.dmactive || ~dmAuthenticated) {
        hgFired(hg) := false.B
    }.elsewhen (~hgFired(hg) & (hgHartFiring(hg) | hgTrigFiring(hg))) {
        hgFired(hg) := true.B
    }.elsewhen ( hgFired(hg) & hgHartsAllHalted(hg) & hgTrigsAllAcked(hg)) {
        //hjr if all firing&halting of the extrigger&harts inside a halt group has been acked:
        //hjr 1:for exttrigger: acked by the outer platform ->extTrigger.out.ack.asBools
        //hjr 2,for harts: halt acked by debug module, that is the ROM code writes its hartid to HALTED,
        //hjr  then combinationally asserts corresponding bit in haltedBitRegs
        //hjr if all firing and halting has been acked, then corresponding hgFired bit will be deasserted.
        hgFired(hg) := false.B
    }
    }

One thing to note about code above is that when all harts halted (hgHartsAllHalted(hg) := (haltedBitRegs.asBools | hgParticipateHart.map(_ =/= hg.U)).reduce(_ & _)) and all external triggers's firing has been acked by the platform(hgTrigsAllAcked(hg) := (trigOutAck | hgParticipateTrig.map(_ =/= hg.U)).reduce(_ & _)), we need to clear that specific halt group's hgFired indicator, marking end of one round halt group firing.

When bit in hgFired for a specific group is set, we need to send halt request to all the harts in that group, even including the hart that caused hgFired to be set, sending debug interrupt again to a hart when it's in debug mode(reg_debug) will be masked:

//sending debug interrupt again to a hart when it's in debug mode(`reg_debug`) will be masked
io.interrupt := (anyInterrupt && !io.singleStep || reg_singleStepped) && !(reg_debug || io.status.cease)


// For each hg that has fired, assert debug interrupt to each hart in that hg
for (component <- 0 until nComponents) {
    hgDebugInt(component) := hgFired(hgParticipateHart(component))
}

We also need to notify all the external triggers in that fired(hgFired) group the firing action:

// For each hg that has fired, assert trigger out for all external triggers in that hg
io.extTrigger.foreach {extTrigger =>
    val extTriggerOutReq = RegInit(VecInit(Seq.fill(cfg.nExtTriggers) {false.B} ))
    for (trig <- 0 until nExtTriggers) {
    extTriggerOutReq(trig) := hgFired(hgParticipateTrig(trig))
    }
    extTrigger.out.req := extTriggerOutReq.asUInt()
}

As a summary, note the circumstances that may cause a hart to halt: 1,when a hart's halt group fired; 2, a hart just snaps out of reset, and halt-on-reset fuctionality takes effect on that hart; 3,an explicit halt request by debugger; 1 and 2 are handled in inner, and hgDebugInt is sent to outer(I have trouble understanding why there is no crossing domain logic needed for this signal???), io.hgDebugInt := hgDebugInt | hrDebugInt, this signal along with 3(explicit hart request) will drive bundle of diplomatic node intnode in outer, and therefore send interrupt request for each hart. Note that intnode in outerAsync is the corss-domain version of that node in outer

//--------------------------------------------------------------
// Interrupt Registers
//--------------------------------------------------------------
val debugIntNxt = WireInit(VecInit(Seq.fill(nComponents) {false.B} ))
val debugIntRegs = RegNext(next=debugIntNxt, init=0.U.asTypeOf(debugIntNxt)).suggestName("debugIntRegs")

debugIntNxt := debugIntRegs

val (intnode_out, _) = intnode.out.unzip
for (component <- 0 until nComponents) {
    /*
    * hjr debug interrupt may happen by
    * 1,debugger writing to haltreq
    * 2,io.hgDebugInt(component)
    *   2.1: reset of a 'hart-on-reset' hart(this dmi register lives inside DMInner)
    *   2.2: a hart or ext trigger fires inside a haltgroup.(DMI register dmcs2 is for handling haltgroup, it lives inside the DMInner )
    *
    * */
    intnode_out(component)(0) := debugIntRegs(component) | io.hgDebugInt(component)
}

//hjr In TLDebugModuleOuterAsync, dmOuter works under the dmi clockdomain, the intnode will go to clock domain of harts, therefore
//hje clock domain corssing is needed. 
val intnode = IntSyncCrossingSource(alreadyRegistered = true) :*= dmOuter.intnode

//hjr signal represented by this diplomatic node will then go to all the tiles in the system:
// 1. Debug interrupt is definitely asynchronous in all cases.
domain.tile.intInwardNode :=
    context.debugOpt
    .map { domain { IntSyncAsyncCrossingSink(3) } := _.intnode }
    .getOrElse { NullIntSource() }

One last thing to note in terms of halt group is that the group 0 is specical, according to the spec: In both halt and resume groups, group 0 is special. Harts in group 0 halt/resume as if groups aren’t implemented at all.When the DM is reset, all harts must be placed in the lowest-numbered halt and resume groups that they can be in. (This will usually be group 0.) In RC impl, all harts and external triggers are in group 0 right after snap out of reset. Consequently, we only set range 1 to nHaltGroups of hgHartFiring,hgTrigFiring and hgFired.

• ABSTRACTCS(particularly cmderr)
  This DMI register contains info about size of program buffer and data section in word unit of the debug module(ABSTRACTCSReset.datacount := cfg.nAbstractDataWords.U and ABSTRACTCSReset.progbufsize := cfg.nProgramBufferWords.U), busy field indicates whether a command is executing (abstractCommandBusy := (ctrlStateReg =/= CtrlState(Waiting))), the cmderr is used for recording the executing status of an abstract command, it will get set if an abstract command fails, it's worth noting that this field is W1C(writing 1 to clear). Refer to the 3.15.6 of debug spec for detailed depiction. The related code is as follows:

//hjr exception is asserted by debug rom code wrting to EXCEPTION 
val errorException   = WireInit(false.B)
//hjr abstract command state machine 
}.elsewhen (ctrlStateReg === CtrlState(Exec)) {

    // We can't just look at 'hartHalted' here, because
    // hartHaltedWrEn is overloaded to mean 'got an ebreak'
    // which may have happened when we were already halted.
    when(goReg === false.B && hartHaltedWrEn && (hartSelFuncs.hartIdToHartSel(hartHaltedId) === selectedHartReg)){
    //hjr an ebreak inside debug mode is met, just jump to ROM start, and set the state machine to be Waiting
    ctrlStateNxt := CtrlState(Waiting)
    }
    when(hartExceptionWrEn) {
    //hjr todo why hartExceptionId === 0.U??
    assert(hartExceptionId === 0.U, "Unexpected 'EXCEPTION' hart")//Chisel3 #540, %x, expected %x", hartExceptionId, 0.U)
        ctrlStateNxt := CtrlState(Waiting)
    errorException := true.B
    }
}.

errorBusy := (ABSTRACTCSWrEnMaybe    && ~ABSTRACTCSWrEnLegal)        ||//hjr try to write ABSTRACTCS
            (autoexecdataWrEnMaybe  && ~ABSTRACTAUTOWrEnLegal)      ||//hjr try to write autoexecdata
            (autoexecprogbufWrEnMaybe && ~ABSTRACTAUTOWrEnLegal)    ||//hjr try to write autoexecprogbuf
            (COMMANDWrEnMaybe       && ~COMMANDWrEnLegal)           ||//hjr try to write COMMAND
            (dmiAbstractDataAccess  && ~dmiAbstractDataAccessLegal) ||//hjr read or write data registers.
            (dmiProgramBufferAccess && ~dmiProgramBufferAccessLegal)//hjr read or write program buffer registers.

when (~io.dmactive || ~dmAuthenticated) {
    ABSTRACTCSReg := ABSTRACTCSReset
}.otherwise {
    when (errorBusy){
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrBusy.id.U
    }.elsewhen (errorException) {
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrException.id.U
    }.elsewhen (errorUnsupported) {
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrNotSupported.id.U
    }.elsewhen (errorHaltResume) {
    ABSTRACTCSReg.cmderr := DebugAbstractCommandError.ErrHaltResume.id.U
    }.otherwise {
    when (ABSTRACTCSWrEn){
        //hjr cmderr is a RW1C, which means:Used to describe a Read/Write-1-to-Clear register field
        //hjr it means that writing 1 for each bit of this field will actually make it 0;
        ABSTRACTCSReg.cmderr := ABSTRACTCSReg.cmderr & ~(ABSTRACTCSWrData.cmderr);//hjr this is for clearing the error.
    }
    }
}

// For busy, see below state machine.
val abstractCommandBusy = WireInit(true.B)//hjr todo why init is true.B
ABSTRACTCSRdData.busy := abstractCommandBusy
when (~dmAuthenticated) {   // read value must be 0 when not authenticated hjr todo then why only datacount and progbufsize is reset.
    ABSTRACTCSRdData.datacount := 0.U
    ABSTRACTCSRdData.progbufsize := 0.U
}

• ABSTRACTAUTO
  This register is a useful proxy for executing brust access, refer to 3.15.8 for further detial. In short words, when one bit in autoexecprogbuf or autoexecdata is set, access to the corresponding word in program buffer or data section will cause the DM to act as if the current value in command was written there again . Therefore, the ABSTRACT and possible instruction in program buffer will execute again using the new value in data section or program buffer. The code sequence is as follows:

//---- ABSTRACTAUTO

val ABSTRACTAUTOReset     = WireInit(0.U.asTypeOf(new ABSTRACTAUTOFields()))
val ABSTRACTAUTOReg       = Reg(new ABSTRACTAUTOFields())
val ABSTRACTAUTOWrData    = WireInit(0.U.asTypeOf(new ABSTRACTAUTOFields()))
val ABSTRACTAUTORdData    = WireInit(ABSTRACTAUTOReg)

val ABSTRACTAUTORdEn = WireInit(false.B)
val autoexecdataWrEnMaybe = WireInit(false.B)
val autoexecprogbufWrEnMaybe = WireInit(false.B)

val ABSTRACTAUTOWrEnLegal = WireInit(false.B)

when (~io.dmactive || ~dmAuthenticated) {
    ABSTRACTAUTOReg := ABSTRACTAUTOReset
}.otherwise {
    when (autoexecprogbufWrEnMaybe && ABSTRACTAUTOWrEnLegal) {
    ABSTRACTAUTOReg.autoexecprogbuf := ABSTRACTAUTOWrData.autoexecprogbuf & ( (1 << cfg.nProgramBufferWords) - 1).U
    }
    when (autoexecdataWrEnMaybe && ABSTRACTAUTOWrEnLegal) {
    ABSTRACTAUTOReg.autoexecdata := ABSTRACTAUTOWrData.autoexecdata & ( (1 << cfg.nAbstractDataWords) - 1).U
    }
}

val dmiAbstractDataAccessVec  = WireInit(VecInit(Seq.fill(cfg.nAbstractDataWords * 4) {false.B} ))
dmiAbstractDataAccessVec := (dmiAbstractDataWrEnMaybe zip dmiAbstractDataRdEn).map{ case (r,w) => r | w}

val dmiProgramBufferAccessVec  = WireInit(VecInit(Seq.fill(cfg.nProgramBufferWords * 4) {false.B} ))
dmiProgramBufferAccessVec := (dmiProgramBufferWrEnMaybe zip dmiProgramBufferRdEn).map{ case (r,w) => r | w}

val dmiAbstractDataAccess  = dmiAbstractDataAccessVec.reduce(_ || _ )
val dmiProgramBufferAccess = dmiProgramBufferAccessVec.reduce(_ || _)

// This will take the shorter of the lists, which is what we want.
val autoexecData  = WireInit(VecInit(Seq.fill(cfg.nAbstractDataWords) {false.B} ))
val autoexecProg  = WireInit(VecInit(Seq.fill(cfg.nProgramBufferWords) {false.B} ))
/*
    * hjr one thing note here:
    * dmiAbstractDataAccessVec has finer granularity(cfg.nAbstractDataWords * 4) than autoexecdata(cfg.nAbstractDataWords), but if a word is accessed, the
    * first byte of that word must be accessed. That is dmiAbstractDataAccessVec(i * 4) could be used as a representation of  accessability of the whole word.
    * */
    (autoexecData zip ABSTRACTAUTOReg.autoexecdata.asBools).zipWithIndex.foreach {case (t, i) => t._1 := dmiAbstractDataAccessVec(i * 4) && t._2 }
    (autoexecProg zip ABSTRACTAUTOReg.autoexecprogbuf.asBools).zipWithIndex.foreach {case (t, i) => t._1 := dmiProgramBufferAccessVec(i * 4) && t._2}
/*
* hjr if a data or prog register is accessed while this register has corresponding autoexecdata or autoexecprogbuf bit set in the
* abstractauto dmi register, this indicates a new command write.
*
* */
val autoexec = autoexecData.reduce(_ || _) || autoexecProg.reduce(_ || _)

autoexec indicates a word in program buffer or data section that corresponds the set bits in autoexecprogbuf or autoexecdata is accessed through dmi bus(it's worth noting that the program buffer and data section also are memory-mapped in the hart's address space, but here only access from dmi bus is considered.dmiAbstractDataAccessVec := (dmiAbstractDataWrEnMaybe zip dmiAbstractDataRdEn).map{ case (r,w) => r | w} and dmiProgramBufferAccessVec := (dmiProgramBufferWrEnMaybe zip dmiProgramBufferRdEn).map{ case (r,w) => r | w}); This will assert val regAccessRegisterCommand = autoexec && commandRegIsAccessRegister && (ABSTRACTCSReg.cmderr === 0.U) if no previous err are set, and therefore starts another round of command execution(transfer the state machine to Exec);

val regAccessRegisterCommand = autoexec    && commandRegIsAccessRegister && (ABSTRACTCSReg.cmderr === 0.U)
//------------------------
// Variable ROM STATE MACHINE
// -----------------------
// hjr Waiting->CheckGenerate->Exec
when (ctrlStateReg === CtrlState(Waiting)){
    //hjr if an actual command write is happening, or a data or progrom buffer register is accessed(the corresponding word is marked in abstractauto),
    // then the command writen process will begin.
    when (wrAccessRegisterCommand || regAccessRegisterCommand) {
    ctrlStateNxt := CtrlState(CheckGenerate)
    }.elsewhen (commandWrIsUnsupported) { // These checks are really on the command type.
    errorUnsupported := true.B
    }.elsewhen (autoexec && commandRegIsUnsupported) {
    errorUnsupported := true.B
    }
//hjr todo have trouble figuring out why this CheckGenerate stata is needed?
}

• CUSTOM register mapping
  The command itself in RC can now only access registers(quick access and access memory command is not supported). However, RC command not only support general purpose register access, it can also support custom register access. Diplomatic sink nodeval customNode = new DebugCustomSink() in TLDebugModuleInner is used for this purpose. Some implementation dependent node can connect to this via debugCustomXbarOpt in HasPeripheryDebug. The bundle behind this diplomatic binding is as follows:

class DebugCustomBundle(val p: DebugCustomParams) extends Bundle {
    val addr = Input(UInt(log2Up(p.addrs.foldLeft(0){_ max _}).W))
    val data = Output(UInt(p.width.W))
    val ready = Output(Bool())
    val valid = Input(Bool())
}

When an abstract command intends to conduct a custom register read(indicated by accessRegIsCustom--just checks whether the regno is in the range of customP.addrs; Also note that addr and valid as output request from debug module, and data and ready as response from the platform), the state machine will transfer to Custom from CheckGenerate, in state Custom wire goCustom will be asserted, and this will send a custom regiser access request to the the platform:

elsewhen (ctrlStateReg === CtrlState(Custom)) {
    assert(needCustom.B, "Should not be in custom state unless we need it.")
    goCustom := true.B
    val (custom, customP) = customNode.in.head
    //hjr goCustom being true.B will combinationally assert the custom.valid, note that customNode in this DM is sink, the custom.valid and custom.addr
    //hjr is output in terms of this DM, the custom.ready and custom.data is input; This directionality is guaranteed by diplomacy.
    /*
    * hjr when an abstract command intends to conduct a custom register read(the only currently supported operation for custom register), the state machine
    * will be in state of Custom, this will send a request along the custom bundle, when the response is ready with valid data, the dmi data register
    * will be updated using the custom data:
    * when (custom.ready && custom.valid) {
    *  (abstractDataMem zip custom_bytes).zipWithIndex.foreach {case ((a, b), i) =>
    *    a := b
    *  }
    * }
    * */
    when (custom.ready && custom.valid) {
    ctrlStateNxt := CtrlState(Waiting)
    }
}
//--------------------------------------------------------------
// Drive Custom Access
//--------------------------------------------------------------
if (needCustom) {
    val (custom, customP) = customNode.in.head
    custom.addr  := accessRegisterCommandReg.regno
    custom.valid := goCustom
}

When the response is ready with valid data, the dmi data register will be updated:

if (needCustom) {
    val (custom, customP) = customNode.in.head
    require(customP.width % 8 == 0, s"Debug Custom width must be divisible by 8, not ${customP.width}")
    val custom_data = custom.data.asBools
    val custom_bytes =  Seq.tabulate(customP.width/8){i => custom_data.slice(i*8, (i+1)*8).asUInt}
    when (custom.ready && custom.valid) {
    (abstractDataMem zip custom_bytes).zipWithIndex.foreach {case ((a, b), i) =>
        a := b
    }
    }
}

Also, the state machine should transfer back to Waiting when the custom bundle fires during the state of Custom, marking end of a custom register access:

    when (custom.ready && custom.valid) {
    ctrlStateNxt := CtrlState(Waiting)
    }

• The holistic hierachy of debug module
  For now, consider the debugger connects to DM via JTAG DTM;
  Inside the Debug Module, there are two sub-components TLDebugModuleInnerAsync and TLDebugModuleOuterAsync, TLDebugModuleInnerAsync and TLDebugModuleOuterAsync works at different clock domains. TLDebugModuleOuterAsync works at the dmiClock domain, that is the TCK fed into the DM from SimJTAG. TLDebugModuleOuterAsync exists so that some debugging operation can be conducted even when the hart is not ready, This allows DMCONTROL.haltreq, hartsel, hasel, hawindowsel, hawindow, dmactive, and ndreset to be modified even while the Core is in reset or not being clocked. Therefore this Outer part is reasonable to function under the outer clock: that is dmiClock;
  The inner part is for handling other dmi register except the ones in outer part, these dmi register is only effective once the harts being debugged snap out of reset and are normally clocked; Moreover, some signal inside the inner part is also memory mapped into the hart's memory space so that the hart could be aware of the debugger request(like FLAGS[hartid].go); Therefore the inner part works at the clock domain of tl_clock, coming from the bus where debug module is connected as a slave, and hart is the master. (There are actually 2 clock domains for the inner part, one is debug_clock, the other is tl_clock, I have trouble figuring out the connection between the two, see this question I posted on RC disscussions which is not being answered yet, but orignal comment somewhere in RC says that these 2 clock domains are synced)
  The original DMI request comes out of DTM, then feeds into the outer part, there is a dmi2tlOpt utility inside TLDebugModuleOuterAync converting dmi request to TL-UL to take advantage of TLRegisterNode functionality(TLRegisterNode is intensively used in RC, refer to chipyard documentation 2.9.4 Register Router for further detail), dmi2tlOpt goes to a crossbar(dmiXbar) diplomatically, (node of) dmiXbar will flow downwards to 2 other nodes: 1, it will go to the dmiNode inside synced version of outer(it's worth noting that both the inner and outer has an async wrapper that handles the clock domain crossing stuff, for example, shift-register an async signal and then feed that signal into the synced version), dmi registers like DMI_DMCONTROL, DMI_HARTINFO, DMI_HAWINDOWSEL, DMI_HAWINDOW are encapsulated into this TLRegisterNode; 2, dmiXbar in TLDebugModuleOuterAync will cross clock domain, and go to the inner part(val dmiInnerNode = TLAsyncCrossingSource() := dmiBypass.node := dmiXbar.node), all other dmi registers are encapulated inside val dmiNode = dmiXing.node in the synced inner. Other non-diplomatic signals that go from outer(innerCtrl, dmactive, etc. ) to inner or vice versa(hgDebugInt) needs to be wrapped with clock domain utilities. As mentioned above that the rationale for seperating outer from inner is because some functionality of the debug module needs to take effect even though the hart is not normally clocked, consequently the request to these inner dmi registers by debugger will be bypassed under this condition: (dmiBypass.module.io.bypass := ~io.ctrl.dmactive | ~dmactiveAck where dmactiveAck means the dmactive has been acked by harts, indicating the system has been normally clocked). The code base corresponding to these logic is staightfoward, see individual trait or classes for details.
  Also note that as mentioned above the inner part of debug module can be accessed by hart, therefore there is a TLRegisterNode tlNode for this in the inner part. The tlNode is slave of tilelink bus where the hart masters : debug.node := tlbus.coupleTo("debug"){ TLFragmenter(tlbus) := _ }. See structure chart about some key connections among different modules