Script to copy all AllSky images and videos to an FTP server

[chvvkumar]

I wrote this shell script to copy all the images that my AllSky camera takes to my NAS. My reason for this is to save these images to train my ML model better. Thought I would share it in case someone else finds use for it.

Name of the script file sync_allsky.sh

#!/bin/bash

# Configuration
LOCAL_DIR="/home/pi/allsky/images"
REMOTE_SERVER="synology.lan" # Replace with your FTP server name
REMOTE_USER=""  # Replace with your FTP username
REMOTE_PASS=""  # Replace with your FTP password
REMOTE_DIR="/ftphome/allsky"

# Log file
LOG_FILE="/home/pi/allsky_sync.log"

# Function to log messages
log_message() {
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" >> "$LOG_FILE"
    echo "$(date '+%Y-%m-%d %H:%M:%S') - $1"
}

# Function to perform a full recursive synchronization
sync_all_files() {
    log_message "Starting full recursive synchronization from $LOCAL_DIR to $REMOTE_SERVER:$REMOTE_DIR"

    # Create a mirror script for lftp to execute
    MIRROR_SCRIPT="/tmp/lftp_mirror_script.txt"
    cat > "$MIRROR_SCRIPT" << EOF
open -u "$REMOTE_USER","$REMOTE_PASS" "$REMOTE_SERVER"
lcd "$LOCAL_DIR"
cd "$REMOTE_DIR"
mirror --reverse --only-newer --verbose
bye
EOF

    # Execute the lftp mirror command using the script
    log_message "Running mirror operation..."
    lftp -f "$MIRROR_SCRIPT"

    if [ $? -eq 0 ]; then
        log_message "Full synchronization completed successfully"
    else
        log_message "Error occurred during synchronization"
    fi

    # Clean up the temporary script
    rm -f "$MIRROR_SCRIPT"
}

# Function to synchronize only new files (for interval-based syncs)
sync_new_files() {
    log_message "Starting synchronization of new/modified files from $LOCAL_DIR to $REMOTE_SERVER:$REMOTE_DIR"

    # Get list of files modified in the last interval
    local files=$(find "$LOCAL_DIR" -type f -mmin -${SYNC_INTERVAL_MINUTES} | sort)

    if [ -z "$files" ]; then
        log_message "No new or modified files found in the last ${SYNC_INTERVAL_MINUTES} minutes."
        return
    fi

    log_message "Found $(echo "$files" | wc -l) file(s) to synchronize"

    # Upload each file
    for file in $files; do
        local rel_path=$(realpath --relative-to="$LOCAL_DIR" "$file")
        local dir_path=$(dirname "$rel_path")

        log_message "Uploading file: $rel_path"

        # Use lftp to upload the file, creating directories as needed
        lftp -u "$REMOTE_USER,$REMOTE_PASS" "$REMOTE_SERVER" << EOF
        cd "$REMOTE_DIR"
        mkdir -p "$dir_path"
        cd "$dir_path"
        put "$file"
        bye
EOF

        if [ $? -eq 0 ]; then
            log_message "Successfully uploaded: $rel_path"
        else
            log_message "Failed to upload: $rel_path"
        fi
    done

    log_message "Synchronization complete"
}

# Check if the local directory exists
if [ ! -d "$LOCAL_DIR" ]; then
    log_message "ERROR: Local directory $LOCAL_DIR does not exist!"
    exit 1
fi

# Parse command line options
FULL_SYNC=false
SYNC_INTERVAL_MINUTES=5
SYNC_ONCE=false

while getopts "fi:o" opt; do
  case $opt in
    f) FULL_SYNC=true ;;
    i) SYNC_INTERVAL_MINUTES=$OPTARG ;;
    o) SYNC_ONCE=true ;;
    \?) echo "Invalid option -$OPTARG" >&2; exit 1 ;;
  esac
done

# If -f flag is provided, do a full sync
if [ "$FULL_SYNC" = true ]; then
    log_message "Starting full recursive sync (all files and directories)"
    sync_all_files

    # If -o (once) flag is also provided, exit after full sync
    if [ "$SYNC_ONCE" = true ]; then
        log_message "One-time full sync completed. Exiting."
        exit 0
    fi
fi

# If -o flag is provided without -f, do a one-time incremental sync
if [ "$SYNC_ONCE" = true ] && [ "$FULL_SYNC" = false ]; then
    log_message "Performing one-time sync of recent files"
    sync_new_files
    log_message "One-time sync completed. Exiting."
    exit 0
fi

# Start the interval-based sync process
log_message "Starting AllSky sync service with ${SYNC_INTERVAL_MINUTES}-minute interval"

# Main loop - run indefinitely
while true; do
    sync_new_files
    log_message "Sleeping for ${SYNC_INTERVAL_MINUTES} minutes before next sync"
    sleep $(( SYNC_INTERVAL_MINUTES * 60 ))
done

Command-Line Options:

-f: Perform a full recursive sync (all files and directories)
-i <minutes>: Set the sync interval in minutes (default: 5)
-o: Run once and exit (can be combined with other options)

Make script executable:

chmod +x sync_allsky.sh

Initial full sync to copy all existing images:

./sync_allsky.sh -f -o

specify a different interval (e.g., 10 minutes):

./sync_allsky.sh -i 10

Create a systemd unit file for installing this script as a service with the name allsky-sync.service:

[Unit]
Description=AllSky Images FTP Sync Service
After=network.target

[Service]
Type=simple
User=pi
# Run with full sync at startup, then continue with 5-minute interval syncs
ExecStart=/home/pi/sync_allsky.sh -f -i 5
Restart=on-failure
RestartSec=10s

[Install]
WantedBy=multi-user.target

Start the service and set it to run on boot:

sudo cp allsky-sync.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl start allsky-sync.service
sudo systemctl enable allsky-sync.service

Check status of the service:

sudo systemctl status allsky-sync.service

[EricClaeys]

@chvvkumar,
Thanks for doing this.
We could use more people like you with programming skills to help improve Allsky. Interested? We have a loooong "to do" list to choose from.

Are you wanting suggestions on how to improve it or make it more usable by others? This could be a good addition to the allsky-config command for one time bulk transfers.

Did you consider configuring your NAS as a Remote Server so every image is automatically uploaded after it's taken? I'll look at the script more closely when I'm on my PC, but it's possible the Remote Server functionality would eliminate the need to constantly run the script.

[chvvkumar]

@EricClaeys

Thank you for the pointer towards outstanding todo list. I will definitely take a look! I am hesitant to commit to taking this now due to some family health emergencies I have going on in addition to work commitments. But I will see what I can contribute.

I am definitely open to suggestions, especially since I kind of aim to get a crude working version and not too experienced enough to make good/more efficient choices.

As for the remote server on my NAS, that is what I tried first but I could only get the latest image.jpg copied to the NAS over FTP. Not the individual images or the videos.

If you are have the time to take a look, I have attached the logs here
support-4534-20250507224243.zip

My config in allsky settings:

and the testUpload script run:

pi@allskypi5:~/allsky/scripts $  testUpload.sh --server
Test upload to Remote Server  succeeded.
Please remove 'ftphome/allsky/testUpload.sh.txt' on your server.

allsky.log errors:

pi@allskypi5:~/allsky/scripts $ grep ERROR /var/log/allsky.log
2025-05-06T11:19:38.199366-05:00 allskypi5 allsky[580680]:    *** upload.sh: ERROR, unable to log in to astro @ synology.lan.
2025-05-06T11:20:11.354389-05:00 allskypi5 allsky[580827]:    *** upload.sh: ERROR, unable to log in to astro @ synology.lan.
2025-05-06T11:20:44.336879-05:00 allskypi5 allsky[580974]:    *** upload.sh: ERROR, unable to log in to astro @ synology.lan.
2025-05-06T11:21:17.343437-05:00 allskypi5 allsky[581131]:    *** upload.sh: ERROR, unable to log in to astro @ synology.lan.
2025-05-06T11:21:50.414600-05:00 allskypi5 allsky[581267]:    *** upload.sh: ERROR, unable to log in to astro @ synology.lan.
2025-05-07T05:32:39.592520-05:00 allskypi5 allsky[302410]:    *** upload.sh: ERROR, RET=1:
2025-05-07T05:34:16.029462-05:00 allskypi5 allsky[302769]:    *** upload.sh: ERROR, RET=1:
2025-05-07T05:42:14.863739-05:00 allskypi5 allsky[310208]:    *** upload.sh: ERROR, RET=1:

pi@allskypi5:~/allsky/scripts $ date
Wed May  7 22:45:43 CDT 2025

FTP from my Pi to the NAS:

pi@allskypi5:~/allsky/scripts $ ftp synology.lan
Connected to synology.lan.
220 diskstation FTP server ready.
Name (synology.lan:pi): astro
331 Password required for astro.
Password:
230 User astro logged in.
Remote system type is UNIX.
Using binary mode to transfer files.

ftp> ls ftphome
229 Entering Extended Passive Mode (|||55856|)
150 Opening BINARY mode data connection for 'file list'.
drwxrwxrwx   1 anonymous ftp               134 May  7 22:49 allsky
226 Transfer complete.

ftp> ls ftphome/allsky
229 Entering Extended Passive Mode (|||55750|)
150 Opening BINARY mode data connection for 'file list'.
drwxrwxrwx   1 astro    users           37750 May  6 11:40 20250504
drwxrwxrwx   1 astro    users          103030 May  6 11:46 20250505
drwxrwxrwx   1 astro    users          114328 May  7 05:45 20250506
drwxrwxrwx   1 astro    users           85556 May  7 22:35 20250507
-rwxrwxrwx   1 astro    users             148 May  7 05:31 data.json
-rwxrwxrwx   1 astro    users         1622738 May  7 22:49 image.jpg
-rwxrwxrwx   1 astro    users              27 May  7 22:47 testUpload.sh.txt
226 Transfer complete.

[EricClaeys]

Set this to "Yes" and it'll upload each image with "YYYYMMDDHHMMSS" in the name.

[chvvkumar]

Ah that makes sense! Thank you.

[EricClaeys]

@chvvkumar,
Here's a version of your script with some changes and comments.
Let me know if you have any questions.

sync_allsky.sh.txt

[chvvkumar]

I will take a look in depth in a bit but I did skim over the changes. They made sense to my untrained eye (especially the variable limitations for huge number of file lists).

I wrote this script with the intent to also use it with my Astro imaging PC as well, to copy NINA output images and PHD2 logs to my NAS which is why I left it a bit generic (and also the fact that I do not know how to use the existing allsky vars in that script :) so thanks for pointing out how to do that in your script )