Clustering the CM4 as much as possible

[VTHMgNPipola]

I wanted to build a rack mountable cluster of Raspberry Pi's for quite some time, and a few days ago sent an email to Jeff asking questions about it, since he had released a video on Uptime Lab's 1U CM4 cluster blades, and he directed me to here.

Firstly, what is actually necessary to have in such a device? Uptime Lab's blade has an HDMI port, USB ports and NVMe builtin, but I don't think anyone will be using one of these when it inside the enclosure, only for debugging and testing purposes (the NVMe yes, but an entire NVMe drive for a single CM4 seems like a waste). An SD card slot seems nice to have though, but I'm not sure since I don't know if anyone actually uses the Lite version of the CM4.

Then there's networking and power distribution. I want to use a 3.5" HDD tray-like form factor to hold the compute modules, and I can probably fit 4 or 5 of them on each one, with appropriate airflow, but that's 48 or 60 1GbE connections per 2U of rack space, and that's a lot. Because of that I wanted to integrate a switch on each blade, that would output 2 2.5GbE connections. For power distribution each blade would connect to a 12v back-plane and convert to 5v internally.

The problem with integrating a switch with each blade is that, since I only recently started working with electronics and low level computer stuff, I don't know if it can work that way, and even if it did you would need a higher bandwidth switch to support every CM4 at full speed. And of course costs, since a BCM53112 costs $25 for a single piece, supports only 4 1GbE connections, and is out of stock, plus the over $20 for 2 2.5GbE magjacks. If it does work however, the benefits are that even without a 2.5GbE switch you could communicate with 4 or 5 CM4's using a single ethernet connection, and you cut in half (or more) the amount of connections you need for all the compute modules (you could even stack 2 clusters with a 48-port switch in the middle, and connect it with 100Gbps links to somewhere else if you really wanted to).

What are your opinions on this? Do you know of any company that can supply such ethernet switch ICs at a lower cost, or know where I can go look for them? Have any idea on how I can put even more compute modules in a smaller space? Have an opinion on anything else?

Sorry for the lengthy post and bad english, thanks xD

[TheGuyDanish]

So, I'm working on something like this at the moment. I've successfully made a board that connects to a backplane to receive 12v for power. Then converts it to 5v with an onboard regulator.

I also want to expand this by adding UART and USB to the backplane, so it can be controlled centrally, perhaps by another CM4. That way you could expose a console and a way to flash the eMMC of blades (if present).

Networking would be interesting to implement, but I don't have the necessary skill to do it.

I'm currently working on the actual backplane, but I'd be happy to open source my repo if anyone would like to help work with me on this.

With regards to cheaper switch ICs, you could use soemthing like the KSZ9897, which is a 7-port chip from Microchip. About £7.70/piece.

[VTHMgNPipola]

I initially thought of using an edge connector like you did, but from what I read it was more expensive than pin headers, so I just used a 2x4 pin header instead. It is a far worse way to connect a board like this to a back-plane, but considering it will slide on a guide on the server I don't think that's a problem

And about the switch IC, it seems like the KSZ9897 only supports 1GbE ports, so you wouldn't be able to use all the bandwidth of every CM4. Though since they won't be used for data storage and have quite weak processors (but can have a decent amount of RAM) this may not be a problem, but I never tested it so I'm not sure.

[TheGuyDanish]

Using an edge connector like the PCI-e one makes a negligible price difference with JLCPCB. We're talking maybe a dollar or two for five boards. That, and it makes it a lot simpler to design the board it slots into. The board on the chassis side will essentially need to be upright in order to mate properly with the blades. That and the PCI-e connector is pretty common, so getting the actual connector on your board, and the required mating parts is quite easy.

With regards to switching capacity. Getting a switch that would allow full gigabit between all blades is going to be lofty. You will need a more expensive chip, hence why I suggested the KSZ9897 for a prototype. My goal is to put 16 blades in 1U. Were I more qualified, including a 24-port switch IC on the board like a VSC7444 could be a very interesting addition, to allow for 1G and 10G ports on the main board. But given this is difficult to achieve, at best, I'm not too hopeful.

I do want to add a microSD card slot to my blades, but my main thought at the moment is to use onboard eMMC as boot storage, then use something like NFS for networked storage.