Skip to content

italoag/repocloner

BranchesTags

Repository files navigation

repocloner

🚀 A high-performance, concurrent GitHub and Bitbucket repository cloner built with Go

CI Go Report Card License: MIT Go Version

repocloner is a powerful command-line tool designed for efficiently cloning multiple GitHub and Bitbucket repositories concurrently. It features an enhanced terminal UI with real-time progress tracking, structured logging, and intelligent worker pool management.

📖 Versão em Português

✨ Features

  • 🔄 Concurrent Processing: Clone multiple repositories simultaneously using configurable worker pools
  • 📊 Real-time Progress Tracking: Interactive terminal UI with live progress updates
  • 🎯 Smart Filtering: Advanced filtering by language, size, fork status, and update date
  • 📁 Flexible Organization: Support for GitHub (users/orgs) and Bitbucket (users/workspaces)
  • 🛠️ Multiple Interfaces: Choose between CLI and TUI (Terminal User Interface)
  • 📋 Multiple Output Formats: Export repository lists as table, JSON, or CSV
  • 🔐 Token Support: GitHub API token and Bitbucket API token integration with rate limiting
  • 🏗️ Domain-Driven Design: Clean architecture with SOLID principles
  • 📝 Structured Logging: Comprehensive logging with configurable levels
  • 🐳 Docker Support: Ready-to-use Docker images

🚀 Installation

📦 Pre-built Binaries

Download the latest release from the releases page:

# Linux (amd64)
curl -L https://github.com/italoag/repocloner/releases/latest/download/repocloner-linux-amd64.tar.gz | tar xz
sudo mv repocloner /usr/local/bin/

# macOS (amd64)
curl -L https://github.com/italoag/repocloner/releases/latest/download/repocloner-darwin-amd64.tar.gz | tar xz
sudo mv repocloner /usr/local/bin/

# Windows (amd64)
# Download repocloner-windows-amd64.zip and extract to your PATH

🐹 From Source (Go)

# Install with Go (requires Go 1.24.3+)
go install github.com/italoag/repocloner/cmd/repocloner@latest

# Or clone and build
git clone https://github.com/italoag/repoclonerr.git
cd repoclonerr
make build
sudo cp build/repocloner /usr/local/bin/

🐳 Docker

# Pull the image
docker pull ghcr.io/italoag/repocloner:latest

# Run with Docker
docker run --rm -v $(pwd):/workspace ghcr.io/italoag/repocloner:latest clone user octocat

# Create an alias for convenience
echo 'alias repocloner="docker run --rm -v $(pwd):/workspace ghcr.io/italoag/repocloner:latest"' >> ~/.bashrc

📚 Usage

🎯 Quick Start

# Clone all repositories from a GitHub user
repocloner clone user octocat

# Clone GitHub organization repositories (skip forks)
repocloner clone org microsoft --skip-forks

# Clone all repositories from a Bitbucket user
repocloner bitbucket user myusername

# Clone Bitbucket workspace repositories
repocloner bitbucket workspace myworkspace --skip-forks

# List repositories in JSON format
repocloner list user torvalds --format json

# Clone with custom settings
repocloner clone user kubernetes --concurrency 16 --depth 1 --base-dir ./repos

🔧 GitHub Clone Command

Clone repositories from a GitHub user or organization:

repocloner clone [type] [owner] [flags]

Repository Types:

  • user or users - Clone from a GitHub user account
  • org or orgs - Clone from a GitHub organization

Examples:

# Basic user cloning
repocloner clone user octocat

# Organization with custom concurrency
repocloner clone org microsoft --concurrency 8

# Include forks and set custom directory
repocloner clone user torvalds --include-forks --base-dir /tmp/repos

# Clone specific branch with shallow depth
repocloner clone org kubernetes --branch main --depth 5

# Clone with debug logging
repocloner clone user facebook --log-level debug

🪣 Bitbucket Clone Command

Clone repositories from a Bitbucket user or workspace:

repocloner bitbucket [type] [owner] [flags]

Repository Types:

  • user or users - Clone from a Bitbucket user account
  • workspace or workspaces - Clone from a Bitbucket workspace

Authentication: Requires Bitbucket API token:

export BITBUCKET_API_TOKEN=your-api-token

Examples:

# Basic user cloning
repocloner bitbucket user myusername

# Workspace with custom concurrency
repocloner bitbucket workspace myworkspace --concurrency 8

# Include forks and set custom directory
repocloner bitbucket user myuser --include-forks --base-dir /tmp/repos

# Clone specific branch with shallow depth
repocloner bitbucket workspace myws --branch develop --depth 3

# Clone with debug logging
repocloner bitbucket user myuser --log-level debug

Available Flags:

Flag Description Default
--base-dir Base directory for cloning .
--branch Specific branch to clone default branch
--concurrency Number of concurrent workers 8
--depth Clone depth (0 for full history) 1
--include-forks Include forked repositories false
--skip-forks Skip forked repositories true
--token GitHub personal access token $GITHUB_TOKEN
--log-level Log level (debug/info/warn/error) info

📋 List Command

List and filter repositories without cloning:

repocloner list [type] [owner] [flags]

Examples:

# List user repositories in table format
repocloner list user octocat

# Export organization repositories as JSON
repocloner list org microsoft --format json

# Filter by language and size
repocloner list user torvalds --language c --min-size 1000000

# Sort by size and limit results
repocloner list org kubernetes --sort size --limit 20

# Filter by update date
repocloner list user facebook --updated-after 2024-01-01

# Export as CSV for spreadsheets
repocloner list org google --format csv --sort updated

Available Flags:

Flag Description Default
--format Output format (table/json/csv) table
--sort Sort by (name/size/updated) name
--limit Limit number of results unlimited
--min-size Minimum repository size (bytes) 0
--max-size Maximum repository size (bytes) unlimited
--language Filter by programming language all
--updated-after Filter by update date (YYYY-MM-DD) all
--include-forks Include forked repositories false
--skip-forks Skip forked repositories true

⚙️ Configuration

🔑 Authentication

GitHub Authentication

repocloner supports GitHub personal access tokens for higher rate limits and private repository access:

# Set via environment variable
export GITHUB_TOKEN="your_token_here"

# Or pass directly
repocloner clone user octocat --token "your_token_here"

Creating a GitHub Token:

  1. Go to GitHub Settings → Developer settings → Personal access tokens
  2. Generate a new token with repo scope
  3. Copy the token and set it as an environment variable

Bitbucket Authentication

repocloner supports Bitbucket API tokens for authentication:

# Set via environment variable
export BITBUCKET_API_TOKEN="your-api-token"

# Or pass directly
repocloner bitbucket user myuser --bitbucket-api-token "your-api-token"

Creating a Bitbucket API Token:

  1. Go to Bitbucket Settings → Personal Bitbucket settings → API tokens
  2. Create a new API token with Repositories: Read permission
  3. Copy the API token and set it as an environment variable

🎨 Terminal UI Features

When cloning repositories, repocloner provides a rich terminal interface:

  • 📊 Real-time Progress: Live updates on cloning progress
  • ⚡ Throughput Metrics: Current speed and estimated completion
  • 📈 Success/Error Counters: Track successful and failed operations
  • 🎯 Current Operation: See which repository is being processed
  • 📝 Detailed Logging: Comprehensive logs with configurable levels

🗂️ Directory Structure

By default, repositories are cloned with this structure:

./
├── repo1/
├── repo2/
└── repo3/

You can customize the base directory:

# Custom base directory
repocloner clone user octocat --base-dir /home/user/projects

# This creates:
/home/user/projects/
├── repo1/
├── repo2/
└── repo3/

🏗️ Architecture

repocloner is built with clean architecture principles:

cmd/repocloner/           # Application entry point
├── main.go

internal/
├── application/       # Business logic layer
│   ├── services/      # Application services
│   └── usecases/      # Use case implementations
├── domain/           # Core business domain
│   ├── cloning/      # Cloning domain
│   ├── repository/   # Repository domain
│   └── shared/       # Shared domain types
├── infrastructure/   # External concerns
│   ├── concurrency/  # Worker pool management
│   ├── git/          # Git operations
│   ├── github/       # GitHub API client
│   └── logging/      # Structured logging
└── interfaces/       # User interfaces
    ├── cli/          # Command-line interface
    └── tui/          # Terminal user interface

Key Design Principles:

  • Domain-Driven Design: Clear separation of business logic
  • SOLID Principles: Single responsibility, dependency inversion
  • Concurrent Processing: Efficient worker pool implementation
  • Error Handling: Comprehensive error management
  • Testability: Clean interfaces for easy testing

🛠️ Development

📋 Prerequisites

  • Go 1.24.3 or later
  • Git
  • Make (optional, for convenience)

🔨 Building

# Clone the repository
git clone https://github.com/italoag/repoclonerr.git
cd repoclonerr

# Build for current platform
make build

# Build for all platforms
make build-all

# Build static binary
make build-static

🧪 Testing

# Run all tests
make test

# Run tests with coverage
make cover

# Run benchmarks
make bench

# Fast testing during development
make test-fast

🎯 Quality Checks

# Run linting
make lint

# Format code
make fmt

# Run security checks
make sec

# Complete quality workflow
make ci

🐳 Docker Development

# Build Docker image
make docker-build

# Run with Docker
make docker-run

# Push to registry
make docker-push

🤝 Contributing

We welcome contributions! Please see our Contributing Guidelines for details.

📝 Development Workflow

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/amazing-feature
  3. Make your changes following our coding standards
  4. Test your changes: make test
  5. Lint your code: make lint
  6. Commit your changes: git commit -m 'Add amazing feature'
  7. Push to the branch: git push origin feature/amazing-feature
  8. Open a Pull Request

🐛 Bug Reports

When reporting bugs, please include:

  • Operating system and version
  • Go version
  • repocloner version (repocloner --version)
  • Steps to reproduce
  • Expected vs actual behavior
  • Any relevant logs or error messages

💡 Feature Requests

We'd love to hear your ideas! Please open an issue with:

  • Clear description of the feature
  • Use case and motivation
  • Proposed implementation (if you have ideas)

📊 Performance

repocloner is optimized for performance:

  • Concurrent Processing: Configurable worker pools (default: 8 workers)
  • Memory Efficient: Streaming operations where possible
  • Rate Limiting: Respects GitHub API limits
  • Shallow Clones: Default depth of 1 for faster cloning
  • Progress Tracking: Minimal overhead real-time updates

Benchmarks (approximate, varies by network and system):

  • Single Repository: 2-5 seconds
  • Organization (50 repos): 30-60 seconds with 8 workers
  • Large Organization (200+ repos): 2-5 minutes with 16 workers

🔍 Troubleshooting

Common Issues

Authentication Errors:

# Verify your token
curl -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/user

# Check token scopes
repocloner list user octocat --log-level debug

Rate Limiting:

# Use authenticated requests
export GITHUB_TOKEN="your_token_here"

# Reduce concurrency
repocloner clone org large-org --concurrency 4

Network Issues:

# Enable debug logging
repocloner clone user octocat --log-level debug

# Check connectivity
curl -I https://api.github.com

Permission Errors:

# Ensure directory is writable
ls -la $(pwd)

# Use custom directory
repocloner clone user octocat --base-dir /tmp/repos

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

  • Charm for the excellent TUI libraries
  • Cobra for the CLI framework
  • Zap for structured logging
  • Ants for worker pool management

📞 Support

Made with ❤️ by Italo A. G.

📖 Read this in Portuguese

About

Git Complete Repository Cloner

repocloner.italo.xyz

Topics

git clone repository clone-coding repository-clone repository-cloning repo-cloner multi-repository

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases 5

Release v0.6.0 Latest
Sep 24, 2025
+ 4 releases

Packages

 
 
 

Contributors 5

Languages