Examples / integration tests sanity

[lann]

I think it's time to work on a coherent plan for how to organize all the example Wasm projects we have floating around:

• crates/*/tests/* (4)
• crates/http/benches/* (2) mea culpa
• examples/* (12)
• templates/* (3)
• tests/http/* (3)

[lann]

Plus another 13 in spin-kitchensink.

[Mossaka]

Since we have templates now, can we put all examples to templates (don't have to be public to all) and let the integration test runtime generate spin application to tests?

[itowlson]

We don't have the concept of non-public templates and I'm not sure I see mixing "examples" in with templates as positive. The nice thing about examples is that you can just run them and get an app that does something and explore from there. Whereas if you open a template in your editor you get a bunch of nasty syntax errors. If we mixed them in together, how would I know which ones I could run up in situ, and which ones I needed to spin new on?

I'm not keen on mixing tests with examples or templates either. Tests should not change just because we wanted to add something to (or remove something from) a template, and test apps should be able to exercise specific features (including to cause errors) in ways that make them highly unsuitable to be templates. Fundamentally test apps are for machine consumption whereas examples and templates are for human consumption.

I do agree on a coherent plan, but part of that is identifying our different purposes and the different situations under which things will need to change. That will guide us towards a clearer organisation while keeping things resilient to change.

[dicej]

I've been chatting with @lann about this and we agree that the next step is to come up with a concrete proposal to reorganize this (which may or may not involve combining directories and/or moving them to other repos. Once we've done that and built some consensus we can implement it.

[vdice]

Although this issue is a bit dusty, it is still highly relevant to all of the example apps under examples, which aren't covered in CI yet. We should come up with a strategy around automated build (and perhaps run/test) of these to ensure they continue to function. Too intrusive on every PR? Maybe a cron job?

(Note that the go SDK does have some Makefile logic for building the tinygo examples; this however isn't run in any recurrent/automated fashion. edit: some are being run via https://github.com/fermyon/spin/blob/main/sdk/go/integration_test.go#L142-L154)