CM5 Super6C working

[pcnate]

I have swapped 2 of the 6 CM4s in my Super 6C (see #425) to run on CM5. Let me know what commands are typically used to provide details about the health of the OS and compatibility of the hardware and I will post it here.

Important note on the physical mounting. The CM5 official heatsink is not compatible with the Super 6C. There are threaded studs on the Super 6C so you screw down through the CM4 heatsink into the threaded studs. On the CM5 heatsink, there are threaded holes on the bottom side of the heatsink. I ended up using the CM4 heatsink with some extra thermal pads to cover the slightly different layout of hardware.

My setup used some NVMe drives that had the OS installed on them. I simply swapped the CM5 and let the same software bootup. I figured it was worth a shot as I've seen this work before between Pi Zeros and 3 or 4s (can't remember)

[dvdmuckle]

Curious about something, what are you using for power? I have a similar setup with three CM4s and two CM5s (with a third ready to install) that are seeing some high IOWait booting off of the same Inland 256GB SSDs; 15% on the CM5s, and around 2-3% on the CM4s. I'm thinking it may be to do with power, and that the CM5s aren't getting enough power for both the SSDs (running at gen3 speeds) and the CM5s themselves. I'm going to test this with an old PSU soon, as the included adapter only handles around 90W, but I'd like to hear your experience with this as well.

[dvdmuckle]

I just got in an adapter for my PSU and have said PSU installed in my cluster. I'm going to see if this helps the IOWait I've been seeing. There's a good chance it doesn't, from early looking...

[dvdmuckle]

Unsure if you've had similar issues with your SSDs, sounds like you haven't. But I am now using an ATX PSU and the following in my cmdline.txt for my CM5 nodes, and those SSDs and nodes seem a lot more stable now:

nvme_core.default_ps_max_latency_us=0 pcie_port_pm=off pcie_aspm=off

Unsure if all of those things are necessary or not, but it works so I'm a bit hesitant to touch it further.

[pcnate]

what were you using to measure your iowait issues?

[dvdmuckle]

Prometheus + Grafana on a K3s cluster. But I was also using Netdata, which is much easier to install and set up. You could also use top but I like Grafana and Netdata for the historical graph.

[dvdmuckle]

For what it's worth as well, before setting those cmdline args, I was seeing a lot of Longhorn volume rebuilds, which would impact Prometheus and cause it to be non-functional unless I deleted the pod.