Oh you did a better implementation than me? - pipe lib + pipe CLI - you might take my Jaq impl or idea? - And/or queue?

[SamuelMarks]

Your blogpost just appeared in my feed! - Looks cool. Especially like the TLA+ stuff and the shell/F# style | usage.

There are two common ways of thinking of shell piping, IMHO:

0. Each | separated 'thing' is a process
1. Each | separated 'thing' is a task

To elaborate on the latter, if each is a task then you might want storage backing (filesystem, SQLite, PostgreSQL, …) or even explicit queue backing (Kafka, RabbitMQ, …). This would enable tasks to be distributed across multiple computers; and/or be executed in parallel; and/or incorporate elaborate interdependencies (implicit and/or explicit: optional; can_execute_in_parallel; is_end_node).

Finally my implementation—can relicense under CC0 if you prefer—was pretty early stage but still functional down this roadmap:

• https://github.com/SamuelMarks/ai-pipe (library)
• https://github.com/SamuelMarks/ai-pipe-ctl (CLI)

Here's a tiny example JSON https://github.com/SamuelMarks/ai-pipe/blob/bc84256/examples/pipeline6.json#L39 - as you can see it uses the Jaq library—jq reimplemented in Rust—and has an embedded HttpClient (reqwest under the hood like you do) and various asserts for in-built testing.

Anyway, that's my 2¢; keep up the great work 🤗

[soareschen]

Hi @SamuelMarks, sorry for the late reply, and thanks for sharing your work! Certainly, the use case for AI pipe could be an interesting use case for building a CGP-based DSL similar to Hypershell.

Probably the main differences is that your project provides a regular runtime-level DSL, where users can submit different pipelines to the program to execute. On the other hand, the approach used by Hypershell requires you to compile the DSL itself as a Rust program. So depending on the specific use cases, this may or may not match your requirements.

If you implement your DSL using Hypershell's approach, you would implement proc macros to take in a code like pipeline6.json, and translate them into Rust types as abstract syntaxes, which would then be handled by your implementation via CGP.

Hopefully this clarifies the differences.

[SamuelMarks]

Yep I agree

I probably won't maintain the project so you're welcome to take any pieces you like and incorporate it as a top-level feature in Hypershell. HttpClient is something you already have at the top-level IIRC, so my recommendation is to add Jaq also.