Skip to content

masterwishx/ca.mover.tuning

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

695 Commits
.github
.github
 
 
.vscode
.vscode
 
 
archive
archive
 
 
plugins
plugins
 
 
source/ca.mover.tuning
source/ca.mover.tuning
 
 
.gitattributes
.gitattributes
 
 
.gitignore
.gitignore
 
 
README.md
README.md
 
 

Repository files navigation

Mover Tuning

About

This is a simple Unraid plugin that will let you fine-tune the operation of the mover.

  • On scheduled runs of mover
    • Only actually move file(s) if the cache drive is getting full (selectable thresholds) or/and based on files age,size,etc.
    • Optionally don't move if a parity check / rebuild is already in-progress.
    • Optionally validate input filenames to prevent attacks on the filename.
    • Optionally select the file-moving tool between Rsync and Unraid's built-in move utility.
  • Optional ability to completely disable the scheduled runs of mover.
  • Manually executed runs of mover ("Move Now" button) or via command line ("mover start") can follow schedule rules or/and always move all files.
  • Expanded functionality with numerous additional options and settings.

This new fork merge all pull requests after review from R3yn4ld. (cosmetics, merge skipfiletypes from shares and add several feature, as for example automatic age threshold, sanitize input filenames to prevent attacks and compatibility with Unraid 7.x, and other stuff coming.

How it works

First it checks if it's valid for this script run: there must be a cache disk present and an instance of the script must not already be running.

Next, check each of the top-level directories (shares) on the cache disk. For all share with 'Use Cache' setting set to "prefer" or "yes", we use 'find' to create a filtered file list of that share directory. For all share with 'Use Cache' setting set to "only", we use 'du' or 'zfs list' to get total size of that share directory.

The list is sorted by "Use cache", increasing age, pool, and file inode, giving priority for being on cache to "cache only" shares, then "cache prefer" by moving newest from array to cache and older to array, and finally to "cache yes" share by moving only from cache to array. Please note that if age setting is set to something else than "Auto (smart cache)" this script is actually dumb and do not check for size and free space and rely on your own calculations. Files at the top level of the cache or an array disk (i.e not in a share) are never moved.

The list is then passed to original unraid mover. For each file, if the file is not "in use" by any process (as detected by 'fuser' command), then the file is moved, and upon success, deleted from the source disk. If the file already exists on the target, it is not moved and the sourceis not deleted. All meta-data of moved files/directories is preserved: permissions, ownership, extended attributes, and access/modified timestamps. If an error occurs in copying a file, the partial file, if present, is deleted and the operation continues on to the next file.

Changelog

  • 2025.06.07

    • fix: Fixed "mover start" from CLI freezing before the end stage at "resetRunOnceMoverSettings" function. (masterwishx)
    • fix: Fixed the mover process retrieving blank values for parent processes instead of "bash" and "crond" commands.
    • new: Added a time counter feature to calculate the elapsed time during file move operations. This enhancement provides better visibility into the performance of file moves.

  • 2025.06.01

    • new: Added an option "Move files tool" to select the file-moving tool between Rsync and the Unraid move utility, default file-moving tool set to Rsync. (masterwishx)
    • new: Introduced a new debug command "mover debug" to generate a diagnostics package for troubleshooting.
    • fix: Enhanced logging and debug information for file move operations.
    • fix: Updated debug package creation to copy the diagnostics ZIP to the system boot logs directory.

  • 2025.05.23

    • fix: Fixed "mover start" issue that was scheduled without the "start" parameter by unRaid 6.x. (masterwishx)
    • new: Added "Top Folder" option for the Clean empty folders feature to remove top-level empty folders on shares.
    • fix: Removed warning message when thresholds are equal and both set to 0%.
    • fix: Enhanced README and plugin with clearer descriptions of plugin functionality, including expanded options and improved usage explanations.
    • fix: Improved disk validation to ensure only mounted disks matching the required pattern are accepted, with clearer error messages and usage instructions.

  • 2025.05.04

    • fix: Fix for initialized PREFER_MOVINGPCTTHRESHOLD to 0. Thanks to AdamLeyshon for reported this issue. (masterwishx)
    • new: A warning will be added when in test mode and thresholds are either identical or have a small gap between them.
    • new: Debug download added that collect all data from plugin and save them to a file for debugging purposes. Thanks to Rysz from forums for the code example.
    • fix: Fix issue where "shareOverrideConfig" with spaces was not working with "grep".
    • fix: Fix for enabling/disabling Mover running on a schedule.

  • 2025.04.24a

    • fix: version number in default.cfg file. (masterwishx)

  • 2025.04.24

    • fix: Fix cli arguments when running "mover command" in cli mode for pass them to age_mover script or original mover. (masterwishx)
    • new: Add "mover start -e diskX" option for age_mover from original mover for empty an array disk.
    • new: Added "mover reset" command to reset all settings in the plugin. This will delete also override existing settings.
    • new: Added a "Defaults" button in the GUI that resets all settings to their default values. This action triggers "age_mover reset" via "reset.php".
    • fix: Updated plugin version handling to ensure it is stored in the config with quotes and displayed correctly in both the console and logs.
    • fix: Schedule option to force move all files by unraid mover now logs output via syslog instead of being unlogged. Thanks to williechan91 for reported this issue.
    • fix: Fix the calculation of PRIMARYSIZETHRESH in cache prefer cases where the threshold can become negative due to freeing thresholds without moving any files. Thanks to AdamLeyshon for reported this issue.

  • 2025.04.05

    • fix: Fix for Logs parent folder can be empty instead of /tmp when no value is provided. Thanks to niwmik2 from forums for reported this issue. (masterwishx)
    • fix: Fixed an issue where (cache:prefer) was generating unnecessary lists for files that should remain on the primary pool.
    • fix: Only generate updated filtered filelist for (cache:prefer) if we have files on secondary storage.
    • new: Added Help block to the plugin page, including useful links and a donation link for support. Thanks to KluthR from forums for the code example.

  • 2025.03.30

    • new: Added new settings: Logs parent folder, age for mover Log,txt and List files. Thanks to Renegade605 and jimlei from forums for the idea. (masterwishx)
    • fix: Log Mover Tuning plugin actions setting when set to No, will not post to syslog and Mover_tuning_xxx.log file.
    • fix: Added a counter when deleting folders and datasets to speed up the counting of files after moving a large number of files. Thanks to Dor from the forums for the idea.
    • fix: Ensure Folders are always deleted when setting "yes" to clean folders after moving.
    • new: Added debug when deleting folders and datasets in order to get more information about the process.

  • 2025.03.20a

    • fix: Exclude primary storage from find in Move now button in cache:prefer share page , when moving all data from array to cache pool by unraid mover. (masterwishx)

  • 2025.03.20

    • fix: Added zfs cache:only share calculation when share is folder instead of dataset. Thanks to Sak from forums for the bug report. (masterwishx)
    • fix: Added remove potential trailing ( /, *, /* ) characters from the skipped folder path in ignore file list path.
    • fix: Fixed issue when thresholds for cache:prefer was applyed to cache:yes shares. Tnanks to Ichthus and other users from forum for the bug report.
    • new: Added fillup (%) global setting threshold option and override for cache:prefer shares.
    • new: Move now button in cache:prefer share page , moving all data from array to cache pool by unraid mover.

  • 2025.03.11

    • Fixed zfs cache pool percent calculation. Thanks to Renegade605 from forums for the bug report. (masterwishx)
    • Array -> Cache (cache:prefer) now moves data to the cache pool from the array only until reaching a fillup limit of 95%.
      • Thanks to alturismo and Renegade605 from forums, who helped clarify how this feature should work.
      • Disabled override setting for mover tuning if shareUseCache="prefer" (Array->Cache) is set.
    • Format global and share settings help text descriptions add more clear description for mover thresholds.
    • Fixed ignore file list path setting when folder in list file contains ([). Thanks to JamieBriers from forums for the PR and fix.

  • 2025.03.04

    • Added option for users can enable/disable Validation (Sanitize) check for input filenames to prevent attacks future added befor. (masterwishx)
    • Fixed Debug = yes/no instead of 0/1 in mover logs.
    • Fixed primary cache prefer not to move data. Set fixed moving threshold to 99% freeing threshold to 0% for skip moving. when chache is full set freeing to 98% to move some data. Maybe better fix will be later.

  • 2025.03.01

  • 2025.02.24

  • 2025.02.18.1752

  • 2025.02.12.1707

  • 2024.09.05.0222

  • 2024.09.05.0115

  • 2024.08.18

  • 2024.08.17

  • 2024.08.15

    • Fix ignore list reserved space double quoting (Thanks silver226 see forum post) (R3yn4ld)
    • Better empty folder cleaner (R3yn4ld)
      • Rewritten to rmdir parent directory of a moved file if empty (drawbacks: will let multidirectory dirs alive)
      • Added option to enable/disable empty folder cleaner added in Settings UI
      • UI improvements, settings sorted
    • Even Better cache priming (hopefully) (R3yn4ld)
      • Rewriten Ignore filelist from file and filetypes filtering functions (major)
      • Improve calculating size of filtered files and filetypes
      • Update calculation from basic/bc to numfmt. Removed bc option
      • Added verification for not breaking hardlinks when an hardlinked file is filtered
    • Added testmode to cleaning empty folder function, and a min depth of 2.
    • Fix ctime bug (R3yn4ld)

  • 2024.08.12

  • 2024.08.11

  • 2024.08.10

    • Better filtering with ctime=no (R3yn4ld)
    • Improved Synchronization (R3yn4ld):
      • Improve synchronization by looking for files on cache first
      • Do not count synchronized files twice (freeing/priming target were half achieved)
      • Optimize Filtering File and Decision loops regarding Rebalance and Synchronize
    • Moved test mode on top of Mover Tuning Page
    • Add check for primary storage not existing (dust config files)

  • 2024.08.07

  • 2024.08.06

    • Bug fixes (R3yn4ld)
      • Resynchronize not working for share below moving threshold.
      • Internal mover moving files from Secondary to Primary instead of syncing (you may Resynchronize to correct the effect)
    • Added Resynchronize all Primary files to Secondary option (R3yn4ld): Resynchronize all Primary files to Secondary. This will resynchronize the Primary (cached) files on both Primary and Secondary (array) so they are backed up and parity protected. All files will be synchronized again independently of modification time. This can be a long operation. Run-once setting will reset back to No after next run
    • Minor bug fixes and improvements (R3yn4ld)

  • 2024.08.05

    • Rebalance shares (R3yn4ld): Enhance previous "Repair Primary" option. Renamed it "Rebalance shares". This will move files from shares to their primary and secondary storage if spread elsewhere. May imply moving older files from Primary->Secondary or Secondary->Primary if allowed (cache:prefer or cache:yes) to free some space.
    • Bug fixes (R3yn4ld)

  • 2024.08.04

  • 2024.08.01

  • 2024.07.30

  • 2024.07.29

  • 2024.07.10

  • 2024.07.07:

  • 2024-06-30:

  • 2023.12.19 (was not in master branch from hugenbd)

    • 4 changes from Swarles (Swarles)
      • Change "while read" lines in age_mover to "while IFS= read -r" to fix trailing white spaces (Swarles)
      • Fix where sometimes mover would not run to mover.old scrip (Swarles)
      • Log if "share.cfg" doesn't exists to help trouble shooting (Swarles)
      • Check for ca.mover.tuning.cfg file and additional logging. (Swarles)

Installation

You can download and install plugins with Community Apps.

Configuration

You'll find its settings within Settings - Scheduler.

Usage

After installation, the default settings are set so that the plugin will be in test mode. You may check /tmp/ca.mover.tuning/Mover_actions_date.list to see if your settings are correct and if the mover will move/keep/sync the files as expected.

There are several commands that can be launched from terminal or a script: /usr/local/emhttp/plugins/ca.mover.tuning/age_mover start To start age mover (the internal moving engine) with the settings you set in the GUI

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover softstop To cleanly exit loops (Filtering, Deciding, Moving) and age_mover. While moving/syncing, the ongoing file transfer is not interrupted and softstop occurs after actual file operation.

/usr/local/emhttp/plugins/ca.mover.tuning/age_mover stop To kill all the process (can lead to unfinished or corrupted file transferts while moving).

See the Mover Tuning_ thread on the Unraid support forum for more details and discussions.

Thanks

This was originally created by Squid.

@2023 - Updated by hugenbd, with contributions by Castcore, Swarles, Dphelan and Davendsai.

@2024 - Updated by R3yn4ld.

@2025 - Updated by masterwishx.

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

9 stars

Watchers

1 watching

Forks

3 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages

  • Shell 89.4%
  • PHP 10.6%