Getting a MISMATCH when accessing invalid PTE using CBO instructions

[Zain2050]

I am working on the covergroup RV64_CBO_VM and getting a mismatch. When I execute cbo.clean (rs1), where rs1 has VA to an invalid PTE, I get a mismatch where Wally is not faulting at the invalid PTE, but imperas is giving me a Store Page Fault.
You can recreate this using this binary
cbo_vm_exception.elf.gz

# Info 671: 'refRoot/cpu', 0x0000030080000b70(main+370): Supervisor 01060613 addi    x12,x12,16
# Info   MEMX 0x30080000b70 0x80000b70 2 0613
# Info   MEMX 0x30080000b72 0x80000b72 2 0106
# Info   x12 0000000000000000 -> 0000000000000010
# Info 672: 'refRoot/cpu', 0x0000030080000b74(main+374): Supervisor 0017a00f **cbo.clean x15**
# Info   MEMX 0x30080000b74 0x80000b74 2 a00f
# Info   MEMX 0x30080000b76 0x80000b76 2 0017
# Info (OP_NCH) GlobalTime:0.000000 LocalTime:0.000007 Net:refRoot/coverpoint 0 => 4
# Info (OP_NCH) GlobalTime:0.000000 LocalTime:0.000007 Net:refRoot/coverpoint 4 => 3
# Info (OP_NCH) GlobalTime:0.000000 LocalTime:0.000007 Net:refRoot/coverpoint 3 => 0x1D
# Info   MEMRM 0x80008028 0x80008028 8 00000000000000ce L3 (refRoot/cpu/64-bit Supervisor Physical unified)
# Info (OP_NCH) GlobalTime:0.000000 LocalTime:0.000007 Net:refRoot/coverpoint 0x1D => 0xC
# Info (RISCV_PTWE) CPU 'refRoot/cpu': Page table entry V=0 [address=0x28080000400 PTEAddress=0x80008028 access=R]
# Info (OP_NCH) GlobalTime:0.000000 LocalTime:0.000007 Net:refRoot/coverpoint 0xC => 1
# Warning (RISCV_IMA) CPU 'refRoot/cpu' 0x30080000b74 0017a00f cbo.clean x15: Page fault at store/atomic memory operation address (0x28080000414)
# Info   mstatus 0000000a00000080 -> 0000000a00000800 [SD:0 MBE:0 SBE:0 SXL:2 UXL:2 TSR:0 TW:0 TVM:0 MXR:0 SUM:0 MPRV:0 XS:0(Off) FS:0(Off) MPP:0->1 VS:0(Off) SPP:0 MPIE:1->0 UBE:0 SPIE:0 MIE:0 SIE:0]
# Info   mepc 0000030080000b50 -> 0000030080000b74
# Info   mcause 0000000000000000 -> 000000000000000f [Interrupt:0 Code:0(Instruction address misaligned)->15(Store/AMO page fault)]
# Info   mtval 0000000000000000 -> 0000028080000414
# Info 673: 'refRoot/cpu', 0x0000000080001480(Mtrampoline): Machine 1000006f jal     x0,80001580
# Info   MEMX 0x80001480 0x80001480 2 006f
# Info   MEMX 0x80001482 0x80001482 2 1000
# Info (IDV) Instruction executed prior to mismatch '0x30080000b70(): 01060613 addi    x12,x12,16'
# Error (IDV) PC mismatch (HartId:0, PC:0x0000000080001480 Mtrampoline+0):
# Error (IDV) Mismatch 0>
# Error (IDV)   . dut:0x0000030080000b74 
# Error (IDV)   . ref:0x0000000080001480 Mtrampoline+0
# Error (IDV) Insn. bit pattern mismatch (HartId:0, PC:0x0000000080001480 Mtrampoline+0):
# Error (IDV) Mismatch 1>
# Error (IDV)   . dut:0017a00f cbo.clean x15
# Error (IDV)   . ref:1000006f jal     x0,80001580
# Error (IDV) CSR register value mismatch (HartId:0, PC:0x0000000080001480 Mtrampoline+0):
# Error (IDV) Mismatch 2> CSR 300 (mstatus)
# Error (IDV)   . dut:0x0000000a00000080 SD:0 MBE:0 SBE:0 SXL:2 UXL:2 TSR:0 TW:0 TVM:0 MXR:0 SUM:0 MPRV:0 XS:0(Off) FS:0(Off) MPP:0 VS:0(Off) SPP:0 MPIE:1 UBE:0 SPIE:0 MIE:0 SIE:0 (not updated)
# Error (IDV)   . ref:0x0000000a00000800 SD:0 MBE:0 SBE:0 SXL:2 UXL:2 TSR:0 TW:0 TVM:0 MXR:0 SUM:0 MPRV:0 XS:0(Off) FS:0(Off) MPP:1 VS:0(Off) SPP:0 MPIE:0 UBE:0 SPIE:0 MIE:0 SIE:0
# Error (IDV) Mismatch 3> CSR 341 (mepc)
# Error (IDV)   . dut:0x0000030080000b50 (not updated)
# Error (IDV)   . ref:0x0000030080000b74
# Error (IDV) Mismatch 4> CSR 342 (mcause)
# Error (IDV)   . dut:0x0000000000000000 Interrupt:0 Code:0(Instruction address misaligned) (not updated)
# Error (IDV)   . ref:0x000000000000000f Interrupt:0 Code:15(Store/AMO page fault)

[Zain2050]

I started working on finding the root cause of the bug. I observed that during the page table walk a HPTWFaultM was occuring (not TLBPageFault which is supposed to occur after the walk finishes). I tracked it down and found out that its being caused due to PMAStoreAmoAccessFaultM from pmachecker.
(line 79) assign PMAStoreAmoAccessFaultM = (WriteAccessM | (|CMOpM)) & PMAAccessFault;
(line 76) assign PMAAccessFault = SelRegions[0] & AccessRWXC | AtomicAccessM & ~AtomicAllowed;

SelRegions[0] & AccessRWXC seems to be the cause of the bug. During a page table walk, the physical address changes to different values (we get the correct value after the walk finishes). As the PAdr changes to 0x10 during the page table walk, SelRegions[0] is high as that address doesn't lie within any address range.
SelRegions[0] is high during probably every walk, but here AccessRWXC causes a problem
(line 55) assign AccessRWXC = ReadAccessM | WriteAccessM | ExecuteAccessF | (|CMOpM);
During other store & load operations ReadAccessM, WriteAccessM & ExecuteAccessF are low during the walk, but in case of CBO instructions CBOpM is not 0. (CBOpM stays 2 for cbo.flush during entire walk).
This causes PMAAccessFault which causes PMAStoreAMOAccessFaultM which in turn leads to HPTWfaultM and stops the page table walk, therefore not getting pte and not checking permissions, leading to a mismatch.
To confirm this I just removed (|CBOpM) from line 55 and the mismatch went away.

What would be a proper fix for this?

[Zain2050]

I found another bug which I didn't notice earlier. I mentioned that an undesired StoreAMOAccessFaultM during HPTW was causing this bug, but I didn't realize that it wan't causing a trap. It should have given me an mcause of 7 in the mismatch. I wrote some other tests in which PTE was pointing to an address outside of our memory range. I got another mismatch as it didn't trap (expected Store access fault).
In hptw.sv
assign HPTWStoreAmoAccessFault = LSUAccessFaultM & DTLBWalk & MemRWM[0];
We are checking memory write using MemRWM[0], which doesn't seem to be affected by cbo instructions. I think we should also check for (|CMOpM) here.
I added this and got a Store access fault as expected.
assign HPTWStoreAmoAccessFault = LSUAccessFaultM & DTLBWalk & (MemRWM[0] | (|CMOpM));

Bug 1: During HPTW we're getting an unexpected StoreAMOAccessFaultM due to which PTE is not properly fetched and permissions aren't checked
Bug 2: Access Faults are not causing a trap. Therefore, in the first bug we're getting a StoreAMOAccessFault but not jumping to the trap handler

[Zain2050]

I have written all the tests which were in the testplan and some additional tests as well and not getting any mismatch with the following changes
For Bug 1:
line 156 of mmu.sv

assign StoreAmoAccessFaultM = (PMAStoreAmoAccessFaultM | PMPStoreAmoAccessFaultM | AtomicMisalignedCausesAccessFaultM & WriteAccessM) & ~TLBMiss & ~(SelHPTW & (|CMOpM));

For Bug 2:
line 116 and 119 of hptw.sv

  assign HPTWStoreAmoAccessFault = LSUAccessFaultM & DTLBWalk & (MemRWM[0] | (|CMOpM));
  assign HPTWStoreAmoPageFault   = PBMTOrDAUFaultM & DTLBWalk & (MemRWM[0] | (|CMOpM));

These changes have resolved the mismatches, but I am not confident that this is the best approach to fix the bug.

[Zain2050]

Found another mismatch while testing for cbo.zero. According to the spec, zicboz instruction is supposed to fault when dirty bit is 0. But wally wasn't faulting. I think line 115 of tlbcontrol.sv needs to be changed to this

assign PreUpdateDA = ~PTE_A | (WriteAccess | CMOpM[3]) & ~PTE_D;

[davidharrishmc]

That sounds plausible. Go ahead and try and test it and make a PR if that solves it.

…

On Sep 9, 2025, at 5:23 AM, Muhammad Zain ***@***.***> wrote: Zain2050 left a comment (openhwgroup/cvw#1496) <#1496 (comment)> Found another mismatch while testing for cbo.zero. According to the spec, zicboz instruction is supposed to fault when dirty bit is 0. But wally wasn't faulting. I think line 115 of tlbcontrol.sv needs to be changed to this assign PreUpdateDA = ~PTE_A | (WriteAccess | CMOpM[3]) & ~PTE_D; — Reply to this email directly, view it on GitHub <#1496 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR4AA35WRIT5Q4CKQGM4GY33R3BCXAVCNFSM6AAAAACCXN4XJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTENZQGQ3DSMJZHA>. You are receiving this because you are subscribed to this thread.

[Zain2050]

Yes this change is working.
@rosethompson can you review all of the changes that I suggested above. That way I'll push everything altogether.

[jordancarlin]

@Zain2050 It'll be easier to review the changes if you make a PR and then you can modify that PR with additional commits if needed. That way we can see the code changes in context.