🎉 Production-Ready Desktop & Web Application - A comprehensive, enterprise-grade fullstack application for S3 bucket migrations with persistent SQLite database, scheduled migration support, real-time monitoring, and detailed reconciliation tracking. Features a modern React dashboard with TypeScript, node-cron scheduling, dual real-time connections (WebSocket + SSE), and comprehensive migration difference analysis.
If you find this project helpful, you can support me here:
- ☕ Support Me
- 📸 Screenshots
- 🌟 Major Accomplishments
- 🚀 Quick Start
- 📦 Installation Guides
- 🏗️ Architecture
- 🛠️ Development
- 📚 Documentation
- 🌟 Features in Detail
- 🏷️ API Reference
- 🤝 Contributing
- 📄 License
Real-time migration statistics with persistent data and accurate metrics
📊 Main Dashboard - Shows persistent migration statistics with SQLite database, real-time updates via WebSocket+SSE, accurate completion rates, and recent migration activity.
Simple S3 alias setup with connection validation
⚙️ Configuration Tab - Clean interface for adding AWS S3, GCP Cloud Storage, Azure Blob Storage, MinIO, wasabi, and other S3-compatible endpoints (Cloudian S3 Hyperstore, IBM Cloud Object Storage, Huawei S3, Pure Storage flashblade, etc.) with built-in connection testing and alias management.
Comprehensive migration wizard with scheduling capabilities
🛠️ Migration Tab - Enhanced wizard with advanced options (overwrite, preserve, exclude patterns), scheduling capabilities, and dry-run capabilities.
Schedule and manage future migrations
⏰ Scheduled Tab - Complete scheduling system with
node-cronbackend, showing pending migrations with countdown timers, reschedule/cancel options, and automatic execution at specified times.
Complete migration tracking with detailed difference analysis
📚 History Tab - Persistent migration history with status filtering, detailed reconciliation reports showing missing/extra/size differences, and actionable reconciliation modals with file-level details.
In-depth difference analysis with actionable insights
🔍 Reconciliation Modal - Advanced difference analysis showing missing files, extra files, size differences, and other discrepancies with full file paths, sizes, and URLs for manual verification or remediation.
Live monitoring with comprehensive reconciliation and bucket analysis
📊 Logs Tab - Real-time migration monitoring with enhanced logs including detailed reconciliation reports, bucket comparison analysis, file-by-file transfer tracking, and comprehensive error handling.
- Windows: Full desktop app with installer, portable, and ZIP packages
- Cross-platform: Electron-based for Windows, Linux, and macOS
- Professional UI: Modern React interface with real-time updates
- Handles millions of objects efficiently with streaming technology
- 3-tier detection system for accurate object counting
- Database-driven comparison for lightning-fast difference analysis
- Memory-efficient processing for massive S3 buckets
- Docker containers for cloud and server deployment
- Standalone executables for individual workstations
- Web interface for browser-based access
Choose your preferred platform:
- 🪟 Windows - Desktop app with installer, portable, and ZIP options
- 🐧 Linux - AppImage, DEB, RPM, and TAR.GZ packages
- 🐳 Docker - Container deployment for servers and cloud
- ✅ Windows v1.0.0 - Available on GitHub Releases
S3 Migration Scheduler-1.0.0-win-x64.zip(Recommended)S3 Migration Scheduler-1.0.0-win-x64.exe(Installer)- Ready for production use!
- 📅 Advanced Scheduling - Cron-based automation with recurring migrations
- 📊 Real-time Monitoring - Live progress tracking with WebSocket updates
- 🗃️ Large-scale Reconciliation - Efficiently handles millions of objects
- 💾 SQLite Database - Persistent migration history and configuration
- 🔧 Built-in MinIO Client - No external dependencies required
- 🖥️ Cross-platform - Windows, Linux, macOS, and Docker support
- Quick Installation Guide - ZIP, installer, and portable options
- Download from Releases - Ready-to-use packages
- Packaging Guide - For developers
- Troubleshooting - Common issues and solutions
- Installation Guide - AppImage, DEB, RPM, TAR.GZ
- System Service Setup - Run as daemon
- Build from Source - Development setup
- Docker Deployment - Container setup and configuration
- Docker Compose - Development environment
- Production Setup - Production-ready configuration
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Electron UI │ │ Web Browser │ │ Docker Web │
│ (Desktop) │ │ (Development) │ │ (Production) │
└─────────┬───────┘ └─────────┬───────┘ └─────────┬───────┘
│ │ │
└──────────────────────┼──────────────────────┘
│
┌─────────────▼───────────────┐
│ Express Server │
│ • REST API │
│ • WebSocket Server │
│ • Cron Scheduler │
│ • Migration Engine │
│ • Reconciliation Engine │
└─────────────┬───────────────┘
│
┌─────────────▼───────────────┐
│ SQLite Database │
│ • Migration History │
│ • Configuration │
│ • Reconciliation Data │
│ • Large-scale Tracking │
└─────────────┬───────────────┘
│
┌─────────────▼───────────────┐
│ MinIO Client │
│ • S3 Operations │
│ • Multi-cloud Support │
│ • Stream Processing │
│ • Massive Object Handling│
└─────────────────────────────┘
- Node.js 18+
- npm 8+
- Git
# Clone repository
git clone https://github.com/hndrwn-dk/s3-migration-scheduler.git
cd s3-migration-scheduler
# Install dependencies
npm install
# Install client dependencies and build
cd client && npm install && npm run build && cd ..
# Install server dependencies
cd server && npm install && cd ..
# Install electron dependencies
cd electron-app && npm install && cd ..
# Start development servers
npm run dev # Starts both backend and frontend# Build client
cd client && npm run build && cd ..
# Build Windows app (✅ TESTED & WORKING)
cd electron-app && npm run build:win && cd ..
# Build Linux app
cd electron-app && npm run build:linux && cd ..
# Build Docker image
docker build -t s3-migration-scheduler .- Large Scale Reconciliation - ✅ IMPLEMENTED Advanced reconciliation system
- Concurrent Users & Detection - Multi-user management
- Migration Workflow - Process diagrams
- Local Testing Guide - Development and testing
- CI/CD Workflows - GitHub Actions automation
- Docker Deployment - Container deployment guide
- Source/Destination Configuration - Support for any S3-compatible storage
- Object Filtering - Include/exclude patterns for selective migration
- Bandwidth Throttling - Control transfer speed to avoid overwhelming networks
- Error Handling - Automatic retry with exponential backoff
- Progress Tracking - Real-time updates with detailed statistics
- Cron Expressions - Flexible scheduling with standard cron syntax
- One-time Migrations - Immediate execution option
- Recurring Migrations - Daily, weekly, monthly, or custom intervals
- Timezone Support - Schedule migrations in any timezone
- Migration Queuing - Smart queue management for multiple migrations
- ✅ Handles millions of objects efficiently with streaming technology
- ✅ Smart Object Detection - 3-tier approach for accurate object count estimation
- ✅ Streaming Inventory - Memory-efficient processing of large buckets (1M+ objects)
- ✅ Database-driven Comparison - Lightning-fast difference detection using SQL
- ✅ Detailed Reports - Comprehensive reconciliation results with actionable insights
- ✅ Progressive Verification - Checkpoint-based resumable reconciliation
- ✅ Scalable Architecture - Designed for enterprise-scale S3 migrations
- Real-time Dashboard - Live migration status and statistics
- WebSocket Updates - Instant progress notifications
- Detailed Logging - Migration-specific log files
- Error Reporting - Comprehensive error tracking and analysis
- Historical Data - Complete migration history with searchable records
GET /api/migrations # List all migrations
POST /api/migrations # Create new migration
GET /api/migrations/:id # Get migration details
PUT /api/migrations/:id # Update migration
DELETE /api/migrations/:id # Delete migration
POST /api/migrations/:id/start # Start migration
POST /api/migrations/:id/stop # Stop migration
GET /api/health # Health check
// Connect to WebSocket
const ws = new WebSocket('ws://localhost:5000');
// Listen for migration updates
ws.on('migration-update', (data) => {
console.log('Migration progress:', data);
});
// Listen for reconciliation updates
ws.on('reconciliation-update', (data) => {
console.log('Reconciliation progress:', data);
});- 🐛 Report Bugs - Open an issue
- 💡 Request Features - Suggest enhancements
- 📖 Improve Documentation - Help make docs clearer
- 🔧 Submit Pull Requests - Fix bugs or add features
- Fork the repository
- Create a feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request
- ESLint configuration for consistent code style
- Prettier for code formatting
- Jest for unit testing
- Conventional Commits for commit messages
This project is licensed under the MIT License - see the LICENSE file for details.
- 📖 Documentation - Check platform-specific guides in
/docs/ - 🐛 Issues - GitHub Issues
- 💬 Discussions - GitHub Discussions
When reporting issues, please include:
- Operating System and version
- Application version or commit hash
- Error messages and logs
- Steps to reproduce the issue
- Expected vs actual behavior
Built with ❤️ for the S3 migration community