About changing satp and sfence.vma

[LIParadise]

This must be some trivial fact, my search skill being bad, or that I understand nothing in the first place...

Anyway currently the supervisor.adoc states:

Executing an SFENCE.VMA instruction guarantees that any previous stores already visible to the current RISC-V hart are ordered before certain implicit references by subsequent instructions in that hart to the memory-management data structures.

So what are "previous stores"? What makes them "visible"? Store operations preceding the sfence.vma on that HART in program order? Store operations done by other HARTs? What if say we change satp from PPN p to PPN q after which immediately issue sfence.vma with vaddr and asid both zero (assuming virtual memory address implied by q happens to contain sfence.vma at pc): the store operations preceding the write to satp in program order correspond to virtual memory space implied by p, but there's no p to be found anymore except maybe in some remnants of the TLB cache, which is unfortunately probably tainted since the fact we know pc in terms of virtual memory space implied by q being sfence.vma is via translating pc based on q...

Also, how about load operations? Load operations on that HART preceding the sfence.vma in program order should also be done in the virtual address space based on the satp to which that load operation corresponded, no?

Maybe my question could be summarized as this: how does the aforementioned quote differ from this quote (from later part of the same doc)?

SFENCE.VMA orders prior explicit accesses before subsequent implicit accesses, and those implicit accesses are ordered before their associated explicit accesses

But then again there's the question why we at times may issue sfence.vma only after change of satp...

I'm so utterly confused.

[gfavor]

Yes, "Store operations preceding the sfence.vma on that HART in program order".

When satp is changed, all preceding loads/stores in program order are translated using the old satp, and all following loads/stores in program order are translated using the new satp. Assuming TLB entries are tagged with ASID (otherwise the satp write will need to flush the TLB), then there may be multiple TLB entries tagged withthe same VA, but only the one tagged with an ASID matching the current satp.ASID will be used to translate that VA.

An sfence.vma is executed in the context of the current satp MODE and ASID. So if one wants to do an sfence.vma wrt to a new translation context, then it has to follow in program order after the satp write.

[gfavor]

When cpu_0 does an sfence.vma, it doesn't matter what is in cpu_1's satp (nor what was previously in cpu_0's satp before the change to its satp). Whatever stores by both cpu's that are visible to cpu_0 when it executes an sfence.vma, get synchronized by the sfence.vma wrt cpu_0's translation caches.

[LIParadise]

Oh so it's more of saying that after cpu_0 issues sfence.vma, its TLB cache about current satp and ASID are as new as cpu_0 can see, correct?

[aswaterman]

Of course we are being informal here, but I think the answer to your question is “yes”.

[gfavor]

Put better, the sfence.vma ensures that after it has exceuted, any remaining translation cache entries are consistent with what's in memory and as modified by all stores visible to cpu_0 (and any stale translation cache entries have been invalidated).

[LIParadise]

I think I understand now.
Sorry for didn't put together more formal a statement.
Thx for you guy's clarification and help!

[LIParadise]

Uh so it turns out I'm not clear, yet...

Why does the spec says that when it recycles some ASID, it should first change satp to the page table with the recycled ASID, after which execute sfence.vma? In order for the HART to know it should execute sfence.vma, the pc virtual address has to point to it, i.e. this instruction fetch is already implicitly referencing the page table with the ASID, and since this ASID is recycled, there may be stale entries in the TLB with this exact virtual address as pc and ASID, meaning the translation might be wrong, meaning pc might well not point to any valid instruction: the sfence.vma might not be there at all.

In short, when recycling ASID, why is it recommended to sfence.vma after satp modification, rather than before?

[gfavor]

Before an ASID can be recycled, all cached translations based on that "old" ASID must be invalidated. And since an sfence.vma operates using the current satp.ASID, satp.ASID must first be changed to the "old" ASID value to be recycled and then hte sfence.vma can be performed.

Also note that the hardware never "knows" when sfence.vma's need to be executed. That is software's responsibility.

[LIParadise]

sfence.vma operates using the current satp.ASID

Would you help elaborate?
Isn't rs2 meant to help synchronize arbitrary ASID (all entries in cache with that ASID or just one leaf page, depending on rs1) in the TLB? It seems to me that if I were to recycle some ASID, I should first clear all its related entries in the cache via setting rs1 to x0 and rs2 to the ASID to be recycled before switching to a satp with that ASID, for else the implicit reference required by pc may see stale entries?

[gfavor]

I mispoke regarding use of the current ASID. ASID-specific sfence.vma's use the ASID value in rs2.

And by the same token, an OS can invalidate "old" translations associated with an old ASID without first switching satp to that ASID - as described by the spec: "Alternatively, software can execute the same SFENCE.VMA instruction while a different ASID is loaded into satp, provided the next time satp is loaded with the recycled ASID, it is simultaneously loaded with the new page table."

[ved-rivos]

In short, when recycling ASID, why is it recommended to sfence.vma after satp modification, rather than before?

In the recycled case, if the ASID in the satp does not change but the PPN i.e. the page table changes, then it is safer and more robust to switch to the new page table and then issue the sfence.vma. Doing the reverse carries the risk that between sfence.vma and the write of the satp with an updated PPN hardware may cache more PTEs from the old page table in the address translation caches. On the other hand, if the page table being switched-from is has a different ASID than the recycled ASID in question then doing an sfence.vma with the recycled ASID first and then switching to that ASID and its page table would be safe.

there may be stale entries in the TLB with this exact virtual address as pc and ASID, meaning the translation might be wrong, meaning pc might well not point to any valid instruction: the sfence.vma might not be there at all.

The part of the OS that recycles ASID and changes the page tables must be identically mapped into both page tables. This is true even if the ASID was not being recycled. As this part of the OS is identically mapped into both page tables, using either page tables leads to the same result.

[gfavor]

Ved's response provides the more complete picture including how, in practice, an OS should go about reliably and safely recycling an ASID. Generally the OS's "recycling" code would reside in a Global page that is mapped across all processes/ASIDs.

[LIParadise]

I guessed I missed the part where hardware might cache some more entries if we don't sfence.vma after the change to satp. That makes sense.

May I say the spec is implicitly requiring the software (OS) to keep certain promises around satp/sfence.vma here, in particular when changing satp, kernel pc (maybe pc + 4 or pc + 8?) in both virtual memory space should refer to the same physical address, s.t. we may change satp without worrying about fetching from wrong places?

[LIParadise]

Oh, right, one may just go with global pages. Makes sense. Thx for the clear explanations from Ved and gf.

[gfavor]

If your satisfied, please close the issue.