Retire-Cluster

Transform your idle devices into a unified AI work cluster! A distributed system based on TCP socket communication that organizes old phones, tablets, and laptops into a powerful computing cluster.

中文文档

🎯 Overview

Retire-Cluster is an innovative solution for repurposing idle devices, designed for users with multiple unused devices. It transforms "retired" phones, tablets, and old laptops into a unified work cluster, bringing new life to these devices.

The system supports automatic device discovery, real-time status monitoring, intelligent task scheduling, and provides a natural language interface for cluster management.

✨ Key Features

♻️ Device Repurposing: Give new life to old phones, tablets, and laptops
🤖 Intelligent Scheduling: Automatically select the best execution node based on device capabilities
📊 Real-time Monitoring: Heartbeat mechanism ensures real-time status updates with automatic failure detection
🔧 Auto-Discovery: Self-aware system that automatically collects and reports hardware/software information
🔄 Dynamic Scaling: Support hot-plugging of devices without service restart
🛡️ Fault Tolerance: Automatic failure detection and task redistribution
🌍 Cross-Platform: Support for Windows, Linux, macOS, and Android (Termux)
⚡ Task Execution: Distributed task execution with priority queuing and requirement matching
🌐 REST API: Comprehensive HTTP API for programmatic cluster management
🔐 Security: API authentication, rate limiting, and input validation
🔗 Integrations: Support for Temporal, Celery, and custom framework integrations
📦 Easy Installation: Single pip install with automatic configuration

🏗️ Architecture

┌─────────────────────┐         ┌─────────────────────┐
│   Main Node         │         │   Worker Node        │
│                     │◄────────┤                     │
│ ┌─────────────────┐ │         │ ┌─────────────────┐ │
│ │Device Registry  │ │         │ │Device Profiler  │ │
│ │Task Scheduler   │ │         │ │System Monitor   │ │
│ │Heartbeat Monitor│ │         │ │Task Executor    │ │
│ └─────────────────┘ │         │ └─────────────────┘ │
└─────────────────────┘         └─────────────────────┘
        TCP Socket                    TCP Socket

Node Types

Main Node: Cluster coordinator and management center
Worker Node: Task execution units supporting various platforms
Storage Node: Data storage and file services (planned)

🚀 Quick Start

Requirements

Python 3.10+
Network connectivity (all devices on the same network)
Optional: psutil for enhanced system monitoring

Installation

# Basic installation
pip install retire-cluster

# With REST API support
pip install retire-cluster[api]

# With framework integrations
pip install retire-cluster[integrations]

# Full installation with all features
pip install retire-cluster[all]

# Or install from source
git clone https://github.com/4ier/retire-cluster.git
cd retire-cluster
pip install .[all]

Start Main Node

# Start main node with default settings
retire-cluster-main

# Start with custom port and data directory
retire-cluster-main --port 9090 --data-dir /opt/retire-cluster

# Initialize configuration file
retire-cluster-main --init-config

Start Worker Node

# Join cluster with auto-detection
retire-cluster-worker --join 192.168.1.100

# Join with custom device ID and role
retire-cluster-worker --join 192.168.1.100:8080 --device-id my-laptop --role compute

# Test connection only
retire-cluster-worker --test 192.168.1.100

Monitor Cluster Status

# View cluster overview
retire-cluster-status 192.168.1.100

# List all devices
retire-cluster-status 192.168.1.100 --devices

# View specific device details
retire-cluster-status 192.168.1.100 --device worker-001

# Filter devices by role
retire-cluster-status 192.168.1.100 --devices --role mobile

# Output in JSON format
retire-cluster-status 192.168.1.100 --json

Start REST API Server

# Start API server with default settings
retire-cluster-api

# Start with authentication and custom port
retire-cluster-api --port 8081 --auth --api-key your-secret-key

# Connect to specific cluster node
retire-cluster-api --cluster-host 192.168.1.100 --cluster-port 8080

Use REST API

# Check API health
curl http://localhost:8081/health

# Get cluster status
curl http://localhost:8081/api/v1/cluster/status

# List devices
curl http://localhost:8081/api/v1/devices

# Submit a task
curl -X POST http://localhost:8081/api/v1/tasks \
  -H "Content-Type: application/json" \
  -d '{"task_type": "echo", "payload": {"message": "Hello API!"}}'

# Check task status
curl http://localhost:8081/api/v1/tasks/{task_id}/status

Task Execution

# Submit tasks programmatically
from retire_cluster.tasks import Task, TaskRequirements, TaskPriority

# Simple task
task = Task(
    task_type="echo",
    payload={"message": "Hello World!"},
    priority=TaskPriority.NORMAL
)

# Task with specific requirements
compute_task = Task(
    task_type="python_eval",
    payload={"expression": "sum(range(1000000))"},
    requirements=TaskRequirements(
        min_cpu_cores=4,
        min_memory_gb=8,
        required_platform="linux"
    )
)

# Submit to scheduler
task_id = scheduler.submit_task(task)

📖 Documentation

Core Documentation

Architecture Guide - Complete system architecture and design
Deployment Guide - Comprehensive deployment instructions
Web Interface Design - Terminal-style web dashboard
CLI Interface Specification - Command reference
AI Integration Guide - AI-friendly interface patterns

Quick Examples

Start Main Node

# Using Docker (Recommended for production)
git clone https://github.com/yourusername/retire-cluster.git
cd retire-cluster
cp .env.example .env
./docker/deploy.sh

# Or run natively
python -m retire_cluster.main_node --host 0.0.0.0 --port 8080

Start Worker Node

# Download worker script
curl -O https://raw.githubusercontent.com/yourusername/retire-cluster/main/examples/simple_worker_client.py

# Run worker
python simple_worker_client.py \
    --device-id my-device \
    --main-host 192.168.1.100 \
    --role compute

Access Web Dashboard

# Open in browser
open http://your-main-node:5000

# CLI-friendly access (AI/automation)
curl http://your-main-node:5000/text/devices
curl http://your-main-node:5000/api/v1/cluster/status

Platform-Specific Setup

Android (Termux)

# Install Termux from F-Droid
pkg update && pkg install python
pip install psutil requests

# Run worker
python simple_worker_client.py --device-id android-$(hostname) --role mobile --main-host 192.168.1.100

Raspberry Pi

# Install dependencies
sudo apt update && sudo apt install python3 python3-pip -y
pip3 install psutil requests

# Run worker
python3 simple_worker_client.py --device-id rpi-$(hostname) --role compute --main-host 192.168.1.100

NAS Deployment (Docker)

# Synology/QNAP NAS
ssh admin@nas-ip
cd /volume1/docker  # or /share/Container for QNAP
git clone https://github.com/yourusername/retire-cluster.git
cd retire-cluster
./docker/deploy.sh --data-path /volume1/docker/retire-cluster

🔧 Advanced Features

Custom Device Roles

Define custom roles for specialized devices:

# In worker configuration
config.role = "gpu-compute"  # For GPU-enabled devices
config.role = "storage"      # For NAS or storage servers
config.role = "mobile"       # For mobile devices

Device Capabilities

The system automatically detects device capabilities:

Computational: CPU cores, GPU availability
Storage: Available disk space
Services: Network services, automation capabilities

Monitoring & Management

Monitor cluster status in real-time:

# Get cluster statistics
stats = registry.get_cluster_stats()
print(f"Online devices: {stats['online_devices']}")
print(f"Total resources: {stats['total_resources']}")

📊 API Reference

Message Protocol

The system uses JSON-based messaging over TCP sockets:

{
  "message_type": "register|heartbeat|status",
  "sender_id": "device-id",
  "data": {
    // Message-specific data
  }
}

Message Types

register: Device registration with capabilities
heartbeat: Periodic status updates with metrics
status: Query cluster or device status
task_assign: Assign task to device (planned)
task_result: Return task execution results (planned)

🤝 Contributing

We welcome contributions! Please see CONTRIBUTING.md for details.

Fork the repository
Create your feature branch (git checkout -b feature/amazing-feature)
Commit your changes (git commit -m 'Add amazing feature')
Push to the branch (git push origin feature/amazing-feature)
Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Inspired by distributed computing projects
Built with Python's asyncio and socket libraries
Special thanks to the open-source community

📞 Contact

GitHub Issues: Report bugs or request features
Discussions: Join the conversation

🗺️ Roadmap

v1.0.0 (Released)

v1.1.0 (Current)

Task execution framework
Intelligent task scheduling and queue management
Built-in task types (echo, sleep, system_info, etc.)
Task requirements and device capability matching
REST API with comprehensive endpoints
API authentication and security middleware
Complete API documentation and examples
Web dashboard with terminal-style interface
Docker deployment support with automated scripts

v1.2.0 (In Progress)

CLI-style web interface with xterm.js terminal
Multi-format APIs (JSON, CSV, TSV, plain text)
Real-time streaming with Server-Sent Events
AI-friendly interface patterns
Docker containerization with health monitoring
Automated backup and recovery scripts
NAS deployment guides (Synology, QNAP)

v2.0.0 (Future)

Enhanced security with authentication and SSL/TLS
Performance monitoring and metrics dashboard
Advanced task scheduling algorithms
Plugin system for custom task types
Configuration management UI
Multi-language worker support (Node.js, Go)

Framework Integrations (Optional)

Temporal workflow integration support
Celery task queue integration
HTTP bridge for external frameworks
Basic monitoring integration (Prometheus metrics)

Give your idle devices a new purpose! 🚀

Name		Name	Last commit message	Last commit date
Latest commit History 37 Commits
.github/workflows		.github/workflows
configs		configs
docker		docker
examples		examples
retire_cluster		retire_cluster
scripts		scripts
tests		tests
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
CLAUDE.md		CLAUDE.md
CONTRIBUTING.md		CONTRIBUTING.md
Dockerfile		Dockerfile
LICENSE		LICENSE
README.md		README.md
README_CN.md		README_CN.md
deploy_template.sh		deploy_template.sh
docker-compose.yml		docker-compose.yml
entrypoint.sh		entrypoint.sh
requirements-web.txt		requirements-web.txt
requirements.txt		requirements.txt
run-claude-with.ps1		run-claude-with.ps1
setup.py		setup.py

License

4ier/retire-cluster

Folders and files

Latest commit

History

Repository files navigation

Retire-Cluster

🎯 Overview

✨ Key Features

🏗️ Architecture

Node Types

🚀 Quick Start

Requirements

Installation

Start Main Node

Start Worker Node

Monitor Cluster Status

Start REST API Server

Use REST API

Task Execution

📖 Documentation

Core Documentation

Quick Examples

Start Main Node

Start Worker Node

Access Web Dashboard

Platform-Specific Setup

Android (Termux)

Raspberry Pi

NAS Deployment (Docker)

🔧 Advanced Features

Custom Device Roles

Device Capabilities

Monitoring & Management

📊 API Reference

Message Protocol

Message Types

🤝 Contributing

📄 License

🙏 Acknowledgments

📞 Contact

🗺️ Roadmap

v1.0.0 (Released)

v1.1.0 (Current)

v1.2.0 (In Progress)

v2.0.0 (Future)

Framework Integrations (Optional)

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Contributors 2

Uh oh!

Languages

Packages