NodeSync

🚀 A powerful multi-node environment and file synchronization tool

中文文档 | English

NodeSync is specifically designed to solve synchronization issues of conda environments, pip packages, and home directory files between multiple nodes. It supports large-scale high-concurrency synchronization operations and provides complete Linux file operation command support.

Features

✨ Environment Sync: Synchronize conda environments and pip packages across multiple nodes
📁 File Sync: Intelligently sync files and directories to multiple nodes
🐧 Linux Commands: Support for complete Linux file operation commands (cp, mv, rm, ls, cat, grep, find, etc.)
🐍 Conda Management: Complete conda package installation, uninstallation, and list management
⚡ High Performance: Support for high-concurrency operations, suitable for large-scale deployment
🔧 Easy to Use: Simple CLI commands without complex configuration
🛡️ Reliability: Built-in retry mechanism and error handling

Quick Installation

PyPI Installation (Recommended)

# Install directly from PyPI
pip install nodesync

# Verify installation
nodesync --version

Development Installation

# Clone the project
git clone https://github.com/Picrew/nodesync.git
cd nodesync

# Install
pip install -e .

# Or use standard method
python setup.py install

Quick Start

1. Configure Nodes

# Add node configuration
nodesync config add-node node0 192.168.1.100 aiscuser
nodesync config add-node node1 192.168.1.101 aiscuser

# View configured nodes
nodesync config list-nodes

2. Package Management

# pip package installation
nodesync-pip install nvitop uv --env ptca

# conda package management
nodesync-conda-install numpy pandas --env ptca
nodesync-conda-list --env ptca
nodesync-conda-remove old-package --env ptca

3. Environment Synchronization

# Sync ptca environment from node0 to node1
nodesync-env sync ptca --from node0 --to node1

# Sync to all other nodes
nodesync-env sync ptca --from node0 --to all

# Automatically create non-existent environments
nodesync-env sync ptca --from node0 --to node1 --create

4. File Operations

# File synchronization
nodesync-cp sync ~/my_project --to all

# File viewing
nodesync-ls -la ~/workspace
nodesync-cat config.txt
nodesync-tail -n 50 app.log

# File searching
nodesync-grep -i "error" ~/logs/*.log
nodesync-find "*.py" ~/project

# File management
nodesync-mv old_file.txt new_file.txt
nodesync-rm -rf temp_folder/
nodesync-mkdir -p ~/new/project/dir

# Permission management
nodesync-chmod 755 script.sh

5. System Management

# System information queries
nodesync-exec "hostname -I"          # View IP addresses
nodesync-exec "df -h"                # Disk usage
nodesync-exec "nvidia-smi"           # GPU status

# Directory size viewing
nodesync-du -sh ~/workspace ~/data

Complete Command List

🔧 Configuration Management

• nodesync config add-node - Add node
• nodesync config list-nodes - List nodes

🐍 Environment and Package Management

• nodesync-pip install - pip package installation
• nodesync-conda-install - conda package installation
• nodesync-conda-remove - conda package removal
• nodesync-conda-list - List packages
• nodesync-env sync - Environment synchronization

📁 File Operation Commands

• nodesync-cp - File copy/sync
• nodesync-mv - File move/rename
• nodesync-rm - File deletion
• nodesync-ls - File listing
• nodesync-mkdir - Create directory

📄 File Viewing and Searching

• nodesync-cat - View file content
• nodesync-head - View file header
• nodesync-tail - View file tail
• nodesync-grep - Search file content
• nodesync-find - Find files

🔍 System Information

• nodesync-du - View directory size
• nodesync-chmod - Modify file permissions
• nodesync-exec - Execute custom commands

Typical Use Cases

Problem 1: node0 has nvitop and uv packages, but node1 doesn't

# Solution 1: Install directly on all nodes
nodesync-pip install nvitop uv --env ptca

# Solution 2: Install with conda
nodesync-conda-install nvitop --env ptca

# Solution 3: Sync entire environment
nodesync-env sync ptca --from node0 --to node1 --create

Problem 2: View node IP addresses

# View all node IPs
nodesync-exec "hostname -I"

# View specific node IP
nodesync-exec "hostname -I" --nodes node1

Problem 3: Code synchronization and deployment

# Sync code (excluding unnecessary files)
nodesync-cp sync ~/workspace --to all \
  --exclude "*.log" --exclude ".git" --exclude "__pycache__"

# Set execution permissions
nodesync-chmod +x ~/workspace/scripts/*.sh

# View logs
nodesync-tail -n 100 ~/workspace/logs/app.log

Problem 4: System monitoring

# System status check
nodesync-exec "df -h && free -h && uptime"

# Find large files
nodesync-find -size +1G ~/

# Clean temporary files
nodesync-rm -rf /tmp/temp_* ~/.cache/pip/

System Requirements

• Python 3.7+
• Linux/macOS systems
• SSH key authentication configured
• Network-connected multiple nodes

Documentation

• Command Quick Reference - Quick lookup for all commands
• Complete Installation and Usage Tutorial - Detailed installation and usage tutorial
• Detailed Usage Guide - Complete usage instructions
• Quick Start Guide - Quick getting started
• Detailed Installation Documentation - Installation instructions

Contributing

Issues and Pull Requests are welcome!

License

MIT License - See LICENSE file for details