S3 Bucket Sync Tool

A TypeScript tool that downloads all files from an S3 bucket into daily directories for historical tracking.

Configuration

All settings are configured via the .env file. Copy .env.example to .env and fill in your values:

cp .env.example .env

Required Environment Variables

# S3 Configuration
S3_BUCKET_NAME=s3://your-bucket-name/prefix/
S3_BUCKET_ONLY=your-bucket-name
S3_PREFIX=your-prefix/

# AWS Configuration  
AWS_REGION=us-east-1
AWS_ACCOUNT_ID=123456789012

# IAM User Configuration
IAM_USER_NAME=your-iam-user
IAM_USER_ARN=arn:aws:iam::123456789012:user/your-iam-user
IAM_POLICY_NAME=your-s3-policy-name
IAM_POLICY_ARN=arn:aws:iam::123456789012:policy/your-s3-policy-name

# AWS Credentials
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key

# Download Configuration
DOWNLOAD_PATH=downloads

Usage

Daily Sync

Run the sync command to download all files from your S3 bucket:

pnpm run sync

This will:

• Connect to S3 using IAM user credentials from .env
• Download all files from your configured S3 bucket and prefix
• Save them in DOWNLOAD_PATH/YYYY-MM-DD/PREFIX/
• Each day creates a new dated directory for historical tracking

Available Commands

pnpm run sync              # Download all S3 files to today's directory
pnpm run diagnose          # Test AWS credentials and S3 access
pnpm run test-path         # Test different S3 path variations
pnpm run check-policy      # Verify IAM policy content
pnpm run check-permissions # Check IAM user permissions

Setup Instructions

1. Install dependencies:

   pnpm install

2. Configure environment variables: Copy the example environment file and fill in your values:

   cp .env.example .env

3. Create IAM policy: Create an IAM policy with the following permissions (replace your-bucket-name with your actual bucket):

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetObject"
               ],
               "Resource": [
                   "arn:aws:s3:::your-bucket-name",
                   "arn:aws:s3:::your-bucket-name/*"
               ]
           }
       ]
   }

4. Attach policy to IAM user: Attach the policy to your IAM user, then create access keys.

5. Test the setup:

   pnpm run diagnose

Project Structure

s3-bucket-sync/
├── downloads/
│   ├── YYYY-MM-DD/         # Daily directories
│   │   └── prefix/         # Files organized by S3 prefix
│   │       └── file.xlsx   # Downloaded files
│   └── ...
├── src/
│   ├── sync.ts             # Main sync script
│   ├── diagnose.ts         # Diagnostic tools
│   └── ...
├── .env                    # Configuration (copy from .env.example)
├── .env.example            # Example configuration
└── package.json