x64: refactor the representation of *Mem and *MemImm

[abrown]

While reflecting on why #10762 is hard, it seemed that the underlying problem to many ISLE matching issues is our inability to actually inspect the variants of the register specific versions of RegMem and RegMemImm. We had been wrapping instances of these Reg* types, but this change pushes the variants down to: GprMem, GprMemImm, XmmMem, XmmMemAligned, XmmMemImm, and XmmMemAlignedImm. With this, it should be possible to match directly on the variants in ISLE. This does not change any functionality.

[abrown]

Is it now theoretically possible to create a GprMem.Reg in ISLE using the wrong kind of register? If so, we could alter this type to Gpr?

[alexcrichton]

Would it be possible to type this as (Gpr (reg Gpr))? That more closely matches the semantics of the GprMem constructor in Rust

[fitzgen]

Following up from today's Cranelift meeting, where we briefly discussed this PR: I want to +1 Alex's comment here.

We want to maintain the invariant that all Gpr* newtypes can only ever contain GPR registers when they aren't a memory or immediate or whatever. If we expose raw Regs in these enum variants, then users can accidentally (say transitively through a series of abstractions) do something like

if let GprMem::Reg { reg } = &mut gpr_mem {
   std::mem::swap(reg, my_xmm_reg);
}

which would break that invariant.

In order to both allow matching on GprMem et al and preserve that invariant, we need to define new enums (to allow matching) but keep the register variants as Gprs (to preserve encapsulation and the GPR invariant). This would look like the ISLE equivalent of something like this:

enum GprMem {
    Gpr { reg: Gpr },
    Mem { addr: SyntheticAmode },
}

Which is, I believe, what Alex is suggesting just above this comment.

[abrown]

I will say, having tried to use this in #10762, that it doesn't just "solve everything." So we should judge this on whether it will make future matching easier. I know that I've reached for this kind of matching in the past but was frustrated it didn't exist.