Skip to content

How to Setup rclone

Jump to bottom Edit New page
Claude Pageau edited this page Jan 13, 2018 · 172 revisions

NOTICE : Rclone is now the current pi-timolo default installed by pi-timolo-install.sh and is at /usr/bin/rclone. If a previous /usr/local/bin/gdrive File Exists, it Will Remain. If rclone is Not already installed upgrade pi-timolo per pi-timolo-install.sh script. You can also install rclone separately per these Rclone Install Instructions. If you have another project where you want rclone support see my rclone4pi GitHub repository

Introduction

Rclone is a command line program to sync files and directories to and from many remote storage services. This utility can be used to update pi-timolo media folders/files to one or more remote services. For more information about rclone See https://rclone.org and/or https://github.com/ncw/rclone

How to Configure a Remote Storage Service

To configure one rclone remote storage service on a raspberry pi computer follow the steps below. If a service is already installed you will see an interactive configuration menu that allows you to change/remove existing service(s) or add a new one. The example below is for setting up a google drive. For other remote services see listings for other services

Example Steps for Google Drive

  • Open putty SSH login session to Raspberry Pi and execute command below, then follow rclone config interactive prompts. You will be required to have a login account on the remote storage service.

    rclone config

  • Select new, At name> prompt, Enter your choice of service name eg gdmedia

  • At Storage> prompt enter number for google drive

  • Follow prompts and accept defaults where appropriate.

  • At Use auto config? select n

  • rclone will display a long url at link: prompt.

  • On RPI SSH session, click and hold left mouse button and highlight https url after link: (do not press enter key)

  • On computer web browser that is logged into service, right click, paste and go on browser url bar.

  • On computer web browser service security web page, Select user account if appropriate. On service security web page confirm access.

  • On confirmation page a Long verification code will be displayed. Use mouse left button to highlight code then right click to copy verification code.

  • On RPI SSH session at prompt Enter verification code> right click, paste verification code then Enter to accept

  • At team drive? prompt Select n if the drive is not part of a team

  • Review and accept settings when prompted.

Install Details for Other Remote Storage Services

For a Full listing Select Storage Systems pull down menu at top of rclone.org Home Page.

DropBox Setup Details

Note: When setting up rclone for DropBox on a headless RPI, you need to install rclone on a windows computer by downloading the appropriate zip file from https://rclone.org then unzip file. At the rclone unzipped folder press shift and mouse right click at same time then select Open command window here. This should open a command window in the unzipped rclone folder. Make sure you are logged into DropBox from default web browser. On command prompt window execute

rclone authorize "dropbox"

A DropBox web page will open requesting to Allow Rclone Access. After you Allow Access on the web browser DropBox page, the command prompt window will have a long security string. Clicked on the top left corner icon of command window then select Edit, Mark. Highlight the two lines of the security string and Enter to copy. Paste result into a text editor and combine both lines of output into one line. Copy the security string and paste into RPI SSH session at the rclone prompt. Respond to rclone prompts and you should have access to your DropBox drive from the RPI. T

How to Test rclone

To test connection to remote service. Execute the following command where gdmedia: is the name you gave your remote storage service. After a short delay this will list the files and folders in the root of your remote drive. Note Remote Paths are specified as remote:path where remote: is name you gave remote storage service.

List configured remote storage services

rclone listremotes

list All files on remote storage service. Change gdmedia to name you gave remote service

rclone ls gdmedia:

list directories

rclone lsd gdmedia:

To get command help, type rclone with no parameters or rclone --help

Example sync From RPI Folder media/motion To remote service name gdmedia: and media/motion folder

rclone sync -v /home/pi/pi-timolo/media/motion gdmedia:media/motion

IMPORTANT : If the media/motion folder has a large number of files and you just want recent status then edit pi-timolo config.py motionRecentMax= variable to required recent entries. Then rclone the media/recent/motion folder or whatever foldermotionRecentDir = variable is set to.

Sample rclone Scripts

Several sample rclone- sync scripts have been provided with the filename prefix rclone- and .sh extension. You can use or copy one of these bash script files and edit variables for your needs. Refer to rclone --help for command parameters.

rclone --help

IMPORTANT : If you are syncing from several RPI's to the same remote storage service. You should change the variable remoteSyncDir to be unique for each Raspberry Pi computer syncing to the same remote storage service. This will avoid sync conflicts. It is also recommended that you clean remote Trash occasionally per example command below.

rclone cleanup gdmedia:

To copy a sample sync script

cd ~/pi-timolo
cp rclone-sync.sh rclone-mysync.sh
nano rclone-mysync.sh

Edit the variables as desired then ctrl-x y to save and exit nano

Change sync to Something Else

Sync makes the destination folder identical to the source folder. Files that are not in the source folder are sent to the destination remote name trash. You should be able to find them there. Rclone can do many things including copy. type rclone --help for all the commands. You can copy one of the rclone sample batch files and replace the sync command with the copy command.

cd ~/pi-timolo
cp rclone-motion.sh rclone-mo-copy.py
nano rclone-mo-copy.py

Use nano to edit per example below and change sync to copy then ctrl-x y to save and exit.

echo "INFO  : /usr/bin/rclone copy -v $localSyncDir $rcloneName:$remoteSyncDir"
echo "        One Moment Please ..."
/usr/bin/rclone copy -v $localSyncDir $rcloneName:$remoteSyncDir

Test changes

./rclone-mo-copy.sh This should copy only the differences and skip over anything that is already existing. Then change cron to run the new rclone batch file.

Use menubox.sh to customize sync script Files

You can also use menubox.sh to select and edit an existing rclone- sync script and use nano to customize and save it to a new filename. You can also test execution of the script from menubox.sh

How to Automate rclone

To avoid multiple instances of rclone running, a small batch file called rclone-sync.sh will only allow one instance to run. Run commands below to edit the rclone-sync.sh script variables.

cd ~/pi-timolo
nano rclone-sync.sh

Edit variables for source, destination, remote storage name as required ctrl-x y to save and exit nano editor

Test script per commands below. After a short delay you should see sync progress.

./rclone-sync.sh

The best way to automate rclone is to add a crontab entry to run sync script(s) at regular intervals. See example below

sudo crontab -e

Example entries to crontab per below. If using nano ctrl-x y to save changes and exit.

*/5 * * * * su pi -c "/home/pi/pi-timolo/rclone-recent.sh > /dev/null 2>&1"
0 5 * * * su pi -c "/home/pi/pi-timolo/rclone-motion.sh > /dev/null 2>&1"
0 5 * * MON su pi -c "/home/pi/pi-timolo/rclone-cleanup.sh > /dev/null 2>&1"
  • First entry runs rclone-recent.sh every five minutes and make the remote folder identical to the source folder.
  • Second entry runs rclone-motion.sh at 5 am every day.
  • Third entry runs rclone-cleanup.sh once a week at 5 am on Monday.

You can change the frequency to what every you like. Just make sure you leave enough time to upload files.

How to Install rclone (if Not already installed by pi-timolo-install.sh)

Rclone is installed with the pi-timolo-install.sh script. rclone is installed at /usr/bin/rclone. If for some reason you want to run rclone-install.sh separately use the Quick Install curl script per below.

Quick Install (if not already installed by pi-timolo-install.sh)

Step 1 Highlight curl command in code box below using mouse left button. Right click mouse in highlighted area and Copy.
Step 2 On RPI putty SSH or terminal session right click, select paste then Enter to download and run script.

curl -L https://raw.github.com/pageauc/pi-timolo/master/source/rclone-install.sh | bash

This will download latest rclone version and install in /usr/bin/rclone

Manual Install

You can also download it separately using wget and manually install. This allows you to inspect script before executing.

wget -O rclone-install.sh -q --show-progress https://raw.github.com/pageauc/pi-timolo/master/source/rclone-install.sh
chmod +x rclone-install.sh
./rclone-install.sh

Upgrade

To upgrade/repair Rclone simply rerun the Quick Install curl script above, Run the pi-timolo-install.sh curl command or manual install. You can also upgrade pi-timolo from menubox.sh UPGRADE Main Menu pick.

footer

Wiki Main Menu

gdrive is no longer default.

Clone this wiki locally