Singularity

The new pure-go implementation of Singularity provides everything you need to onboard your, or your client's data to Filecoin network, with automatic deal creation and intelligent workflow management.

✨ Key Features

• 🚀 Automatic Deal Creation - Deal schedules created automatically when data preparation completes
• 📦 Data Preparation - Efficient scanning, packing, and CAR file generation
• 🔗 Deal Management - Comprehensive deal scheduling and tracking
• 💼 Wallet Management - Full wallet lifecycle with metadata support for organization
• 🏪 Storage Integration - Support for multiple storage backends (local, S3, etc.)
• 📊 Monitoring & Notifications - Real-time status updates and error handling
• 🔧 Flexible Configuration - Extensive customization options for different workflows

🚀 Quick Start

Installation

# Download the latest release
wget https://github.com/data-preservation-programs/singularity/releases/latest/download/singularity-linux-amd64
chmod +x singularity-linux-amd64
sudo mv singularity-linux-amd64 /usr/local/bin/singularity

# Or build from source
git clone https://github.com/data-preservation-programs/singularity.git
cd singularity
go build -o singularity .

Basic Usage

Single command data onboarding with automatic deal creation:

singularity onboard \
  --name "my-dataset" \
  --source "/path/to/data" \
  --auto-create-deals \
  --deal-provider "f01234" \
  --deal-verified \
  --deal-price-per-gb 0.0000001 \
  --start-workers \
  --wait-for-completion

That's it! ✨ This single command will:

1. Create storage connections automatically
2. Set up data preparation with deal parameters
3. Start managed workers to process jobs
4. Automatically progress through scan → pack → daggen
5. Create storage deals when preparation completes
6. Monitor progress until completion

🤖 Auto-Deal System

The Auto-Deal System automatically creates deal schedules when data preparation jobs complete, eliminating manual intervention. The onboard command provides the simplest interface for complete automated workflows.

How It Works

Source Data → Scan → Pack → DAG Gen → Deal Schedule Created ✅

All stages progress automatically with event-driven triggering - no polling or manual monitoring required.

Configuration Options (onboard command)

Flag	Description	Default
--auto-create-deals	Enable automatic deal creation	true
--deal-provider	Storage provider ID (e.g., f01234)	Required
--deal-verified	Create verified deals	false
--deal-price-per-gb	Price per GB per epoch	0
--deal-duration	Deal duration (e.g., "8760h")	535 days
--deal-start-delay	Deal start delay	72h
--validate-wallet	Validate wallets before creating deals	false
--validate-provider	Validate storage provider	false
--start-workers	Start managed workers automatically	true
--wait-for-completion	Monitor until completion	false

Manual Monitoring

# Check preparation status
singularity prep status "my-dataset"

# List all deal schedules
singularity deal schedule list

# Run background processing service
singularity run unified --max-workers 5

📖 Documentation

Read the Full Documentation

🛠️ Advanced Usage

Multiple Storage Providers

Onboard data to different providers with different strategies:

# Hot storage with fast provider
singularity onboard --name "hot-data" --source "/critical/data" \
  --deal-provider "f01234" --deal-price-per-gb 0.000001 --auto-create-deals

# Cold storage with economical provider  
singularity onboard --name "cold-data" --source "/archive/data" \
  --deal-provider "f05678" --deal-price-per-gb 0.0000001 --auto-create-deals

Conditional Auto-Deals

Use validation to control when deals are created:

# Only create deals if wallet has sufficient balance
singularity onboard --name "conditional" --source "/data" --auto-create-deals \
  --deal-provider "f01234" --wallet-validation

# Only create deals if provider is verified  
singularity onboard --name "verified-only" --source "/data" --auto-create-deals \
  --deal-provider "f01234" --sp-validation

Monitoring

# Check preparation status
singularity prep status "my-dataset"

# List all deal schedules
singularity deal schedule list

# Run unified service with monitoring
singularity run unified --max-workers 5

Error Logging

Singularity automatically logs errors and events during operations. View and query error logs to troubleshoot issues:

# List recent error logs
singularity error log list

# Filter by component and level
singularity error log list --component onboard --level error

# Find errors for specific preparation
singularity error log list --entity-type preparation --entity-id 123

# View only critical errors
singularity error log list --level critical

# Check deal scheduling issues
singularity error log list --component deal_schedule --level warning

# Query logs from the last hour
singularity error log list --start-time $(date -d '1 hour ago' -Iseconds)

# Paginate through results
singularity error log list --limit 20 --offset 40

# Export logs as JSON for analysis
singularity error log list --json --component onboard > error_logs.json

Error Levels:

• info: General information and successful operations
• warning: Non-critical issues that should be monitored
• error: Failures that need attention but don't stop operations
• critical: Severe failures requiring immediate action

Common Components:

• onboard: Data scanning, preparation, and CAR file generation
• deal_schedule: Deal creation and scheduling operations
• pack: Data packing and piece generation
• daggen: DAG generation from packed data

Enhanced State Recovery

Singularity includes an enhanced state recovery system that automatically detects and recovers from missing or inconsistent deal state change records on startup.

How State Recovery Works

1. Automatic Detection: On service startup, the system scans all deals to identify:

   • Deals missing state change records
   • Deals with inconsistent states between the deal table and state change history
   • Deals that may have experienced tracking issues

2. Recovery Actions:

   • Creates initial state change records for deals missing any history
   • Logs detailed information about detected inconsistencies
   • Provides clear remediation guidance in logs

3. Recovery Process:

   # The recovery runs automatically on startup
   singularity run unified
   
   # Look for recovery logs
   tail -f singularity.log | grep "state recovery\|RecoverMissingStateChanges"

Expected Logs

During recovery, you'll see logs like:

INFO: Starting recovery of missing deal state changes
INFO: Creating initial state change for deal 123 (provider: f01234, state: published)
WARN: Deal 456 has inconsistent state - current: confirmed, last tracked: published
INFO: Recovery complete - processed 150 deals, recovered 5, found 2 inconsistencies

Monitoring Recovery

# Check for recovery-related errors
singularity error log list --component deal_state_recovery

# View deals with state issues
singularity deal list --filter "state_inconsistent"

# Check specific deal state history
singularity deal state-changes 123

Remediation Options

If inconsistencies are detected:

1. Review Logs: Check the detailed logs for specific deal IDs and their state discrepancies
2. Manual Verification: Use provider APIs or on-chain data to verify actual deal states
3. Update States: If needed, manually update deal states to match reality:

   singularity deal update 123 --state confirmed

4. Re-run Recovery: The recovery process runs on each startup, ensuring ongoing consistency

The recovery system helps maintain data integrity and ensures accurate deal tracking even after unexpected interruptions or state tracking failures.

🏗️ Architecture

Simplified Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│  Onboard        │    │ Worker Manager  │    │ Workflow        │
│  Command        │────▶│                 │────▶│ Orchestrator    │
│                 │    │ • Auto-scaling  │    │                 │
│ • One command   │    │ • Job processing│    │ • Event-driven  │
│ • Full workflow │    │ • Monitoring    │    │ • Auto-progress │
└─────────────────┘    └─────────────────┘    └─────────────────┘
                                  │                        │
                                  ▼                        ▼
                    ┌─────────────────────────────┐ ┌──────────────┐
                    │     Auto-Deal Service       │ │ Deal Schedule│
                    │                             │ │    Created   │
                    │ • Check Readiness          │ │      ✅      │
                    │ • Validate Wallets/SPs    │ │              │
                    │ • Create Deal Schedules    │ │              │
                    └─────────────────────────────┘ └──────────────┘

Key Components

• Onboard Command: Single entry point for complete automated workflows
• Worker Manager: Auto-scaling workers that process jobs intelligently
• Workflow Orchestrator: Event-driven progression through data preparation stages
• Auto-Deal Service: Creates deal schedules when preparations complete
• Trigger Service: Handles automatic deal creation logic
• Validation System: Ensures wallets and providers are ready for deals
• Notification System: Provides observability and error reporting

🧪 Testing

# Run auto-deal tests
go test ./service/autodeal/ -v

# Run integration tests
go test ./service/autodeal/ -v -run "TestTrigger"

# Test CLI functionality
singularity onboard --help

🔧 Configuration

Environment Variables

# Lotus connection
export LOTUS_API="https://api.node.glif.io/rpc/v1"
export LOTUS_TOKEN="your-token"

# Database
export DATABASE_CONNECTION_STRING="sqlite:singularity.db"

Runtime Configuration

# Run unified service with custom settings
singularity run unified --max-workers 5

# Run with specific worker configuration
singularity run unified --max-workers 10

🚨 Troubleshooting

Common Issues

Auto-deal not triggering:

• Ensure --auto-create-deals is enabled when using onboard
• Verify wallet is attached: singularity prep list-wallets <prep>
• Check all jobs are complete
• Verify unified service is running: singularity run unified

Deal creation failing:

• Check provider ID is correct
• Ensure wallet has sufficient balance
• Verify network connectivity to Lotus
• Review validation settings

Performance issues:

• Adjust --max-workers in unified service for better throughput
• Monitor database performance and connections
• Use appropriate hardware resources for large datasets

Debug Commands

# Test onboard workflow
singularity onboard --name "test-dataset" --source "/test/data" --auto-create-deals

# View detailed logs
singularity run unified --max-workers 3

# Check preparation status
singularity prep status "my-dataset"

🤝 Migration from Manual Workflows

Existing preparations work unchanged! Auto-deal is completely opt-in:

# Existing workflow (still works)
singularity prep create --name "manual"
singularity deal schedule create --preparation "manual" --provider "f01234"

# New automated workflow
singularity prep create --name "automatic" --auto-create-deals --deal-provider "f01234"

📊 Monitoring & Observability

Key Metrics

• Preparations processed per minute
• Deal schedules created automatically
• Validation success/failure rates
• Error frequencies and types

Log Analysis

# Monitor auto-deal activity
tail -f singularity.log | grep "autodeal-trigger\|auto-deal"

# View successful deal creations
grep "Auto-Deal Schedule Created Successfully" singularity.log

🌟 Benefits

Before Auto-Deal System

• ❌ Manual deal schedule creation required
• ❌ Risk of forgetting to create deals
• ❌ No automation for completed preparations
• ❌ Time-consuming manual monitoring

After Auto-Deal System

• ✅ Zero-touch deal creation for completed preparations
• ✅ Configurable validation and error handling
• ✅ Background monitoring and batch processing
• ✅ Comprehensive logging and notifications
• ✅ Full backward compatibility

🔮 Future Enhancements

• Dynamic provider selection based on reputation/pricing
• Deal success monitoring and automatic retries
• Cost optimization algorithms
• Advanced scheduling (time-based, capacity-based)
• Multi-wallet load balancing
• Integration with deal marketplaces

📞 Support

For issues or questions:

1. Check logs: tail -f singularity.log | grep auto-deal
2. Review notifications: singularity admin notification list
3. Run tests: go test ./service/autodeal/ -v
4. Consult documentation: Full Documentation

Related Projects

• js-singularity - The predecessor that was implemented in Node.js
• js-singularity-import-boost - Automatically import deals to boost for Filecoin storage providers
• js-singularity-browser - A next.js app for browsing singularity made deals
• go-generate-car - The internal tool used by js-singularity to generate car files as well as commp
• go-generate-ipld-car - The internal tool used by js-singularity to regenerate the CAR that captures the unixfs dag of the dataset.

License

Dual-licensed under MIT + Apache 2.0

Integration Tests & MongoDB

Some integration tests require a MongoDB instance running on localhost:27018.

• CI: MongoDB is automatically started on port 27018 in GitHub Actions workflows.
• Local Development: You must start MongoDB locally on port 27018 before running tests:

mongod --port 27018

If MongoDB is not available, related tests will be skipped or fail with a connection error.