DAIN – Distributed AI (Training) Network

[secondtruth]

1. Overview

A decentralized network for collaborative AI model training, leveraging distributed computing resources from volunteers worldwide.

This concept for DAIN aims to create a truly decentralized and community-driven approach to AI model development, aligning with OWTA's goals of promoting open (web) technologies and decentralization.

Find the development repository here

1.1 Key Components

• Distributed Computing System powered by BOINC
• PyTorch-based API for Project Definition
• Advanced Training Approaches (Data and Model Parallelism)
• Intelligent Client Clustering and Task Assignment

2. Technical Approach

2.1 Distributed Computing System powered by BOINC

To avoid reinventing the wheel, DAIN will integrate and build upon existing technologies:

BOINC's established distributed computing framework will be leveraged for managing the distribution of AI training tasks across participating machines.

Integration benefits:

• Proven volunteer computing model
• Existing user base familiar with contributing computing power
• Robust infrastructure for managing distributed tasks

2.2 PyTorch-based API

Implementing a PyTorch-based API for defining and managing training projects:

• Definition using hyperparameters
• Starting model (if applicable)
• Model definition
• Custom code support (with security considerations)
• Dataset management
• Job chunking for efficient distribution

3. Training Approaches

Initial Focus: Data Parallelism

• Implement data-parallel training as the primary approach for the first version
• Utilize PyTorch's distributed training capabilities

Future Goal: Distributed Model Parallelism

• Research and develop efficient methods for distributed model-parallel training
• Implement sparse communication between distributed clients
• Design a hierarchical parallelism system:
  • Model-parallel training within clusters of closely located clients
  • Data-parallel training between clusters

4. Client Clustering and Task Assignment

• Develop algorithms for clustering clients based on:
  • Network latency (primary factor for model-parallel groups)
  • Compute speed (secondary factor for model-parallel groups, primary for data-parallel)
• Implement intelligent task assignment considering the hierarchical parallelism structure

5. Governance

• Establish a community-driven process for selecting training projects and voting on model architectures and training objectives
• Implement transparency measures for all relevant aspects of the project and/or form a decentralized autonomous organization (DAO)

6. Next Steps

• Form a working group within OWTA to further develop this concept
• Reach out to potential partners in the AI and distributed computing fields
• Create a detailed technical specification or whitepaper
• Develop a proof-of-concept implementation

7. Attribution and Acknowledgements

The core concept of DAIN was initially proposed by @icec102, who also provided valuable thoughts about potential technical approaches.

8. References and Further Reading

• Getting Started with Distributed Data Parallel – PyTorch
• Accelerate – Hugging Face
• GPU computing – BOINC
• APIs for add-on software – BOINC