Vote: what distributed programming use cases would you like to see?

[pchiusano]

We've planning a series of posts introducing Unison's distributed programming support.

Good news: the API is quite general and capable of handling lots of different use cases!

Bad news: there's too many choices and we're not sure which ones to work up first. This ticket is a place for you to vote informally on which use cases you're most interested in, or to propose other ideas. To vote just use 👍 on any of the ideas below.

You can use comments on this thread for discussion, or if that gets unwieldy, I'd suggest using the #distributed channel of the Slack.

Also, feel free to DM @pchiusano on the Unison Slack or one of the other maintainers to chat about any of these. You can also contact us here. We are very interested in hearing from you!

• Can we write a baby Spark clone in under 100 lines of Unison?
• Can we implement a CDN or IPFS in under 100 lines of Unison?
• Your favorite ETL job, parallel CSV parsing, or S3 bucket downloading
• Resillient scheduled jobs and workflows
• Suffix arrays, search engines, and bioinformatics
• Distributed monte carlo simulation

[pchiusano]

Can we write a baby Spark clone in under 100 lines of Unison?

Show a Spark-like library for creating and transforming distributed datasets, demonstrating a general technique for turning any functional data structure into a distributed version.

Current status: published

[pchiusano]

Could we implement a CDN or IPFS in under 100 lines of Unison?

Work up to a locality-aware, demand-replicated, highly scalable distributed hash table. This is a powerful building block, suitable for implementing a CDN, an IPFS clone, or acting as the source of truth for a large system. How simply can we build one of these in Unison?

[pchiusano]

Your favorite ETL job, parallel CSV parsing, or S3 bucket downloading

For instance, perhaps some ad hoc analysis of a few terabytes of log files, or some ETL job as part of data prep before starting a machine learning task.

Or perhaps something like "load these CSV files, munge them a bit, then put stuff into the database".

[pchiusano]

Resillient scheduled jobs and workflows

A lot of "boring" business workflows have computations that execute over a period of days or weeks, with human intervention at various stages. Think: a user signup flow (which requires first sending an email, then they confirm it, then they provide credit card info, then...) or a recurring billing task ("wake up once a month and bill the user, unless the account has been cancelled"). People often write these workflows using spaghetti code spread across several services, with state manually persisted and unpersisted by different stages of the workflow.

The idea here is that you instead write these workflows using straight line code using an Async ability of some sort. The business logic is nicely consolidated in one place, and Unison takes care of persisting the continuations and resuming the workflow when tasks complete.

There are frameworks for this sort of thing (such as temporal). How simple and general can we make this in Unison?

[pamu]

Workflow orchestrator like Argo/Airflow in Unison.

• Argo Workflows is a Kubernetes-native workflow engine for orchestrating parallel jobs as containers, ideal for CI/CD, data processing, and ML pipelines.
• Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring batch-oriented workflows using Python-defined Directed Acyclic Graphs (DAGs), widely used for data engineering pipelines.

[pchiusano]

Suffix arrays, search engines, and bioinformatics

Show an implementation of distributed suffix arrays, useful for building a full-text search engine and for various bioinformatics applications.

[pchiusano]

Distributed monte carlo simulation

This is a pretty straightforward application. Programs that use randomness can be run with a distributed handler that runs multiple samples in parallel on lots of nodes. Nothing needs to be set up in advance--dependencies will be deployed on the fly.

Likely useful for finance, derivatives pricing, or for certain kinds of machine learning.

[runarorama]

Distributed ledger

Implement a blockchain or distributed ledger as a Unison library.

[francisdb]

Distributed queues and topics

[tscholak]

Distributed matrix multiplication where neither matrix fits on a single node.