Differential Testing for Primitives

[ayakayorihiro]

This post documents an idea that I've discussed with @sampsyo @ekiwi @EclecticGriffin @obhalerao. Please let me know if I'm missing anything!

Currently, we have four implementations of Calyx primitives:

1. the original Verilog primitive
2. the Rust implementation used in Cider
3. the btor2 translation derived from Verilog
4. the FIRRTL implementation used for the firrtl backend

Plus, there can be more in the future!

It would be really nice to have an automated differential testing strategy/harness that checks across these four primitive implementations. This differential testing rig would be especially helpful for any future primitive implementations.

For the record, I've written a quick bash script (currently not executable on its own since I assume a setup that's only on my machine, but will fix that soon) that compares 1 (Verilog) against 4 (FIRRTL) by running an input Calyx program and comparing the outputs via a diff. I imagine we can expand this to (a) an actual rig in Rust/Python(?) instead of some bash I threw together in one sitting, (b) provide randomized inputs.

[sampsyo]

Yes, great! As your current shell script demonstrates, it seems like a productive strategy here (although not the only possible one) would be to automatically generate little Calyx wrappers for each primitive. This would be the "hard part": looking at the primitive's signature and producing a little Calyx program that has enough input/output memories for all of the primitives input/output ports, then generating the logic to drive the primitive from those memories.

Seems like a fun, self-contained little project!

[rachitnigam]

We also had the Calyx fuzzing tool (#889) which got removed in an older PR (#1627). It could provide a potentially better starting point!