Methods to the madness: Copying files into vaults

[ebullient]

Copying files into your vault is a necessary task, but there can be some nuance to it.

I'd like to compare notes!

[ebullient]

I use a script that runs rsync twice: once for notes and once for images:

#!/bin/bash

rsync -vi -rlD -O -c --delete --stats \
    --include "*/" \
    --exclude bestiary/bestiary.md \
    --exclude items/potion-of-resistance.md \
    --exclude items/chromatic-rose.md \
    --exclude items/armor-of-resistance.md \
    --include="*.md" \
    --exclude="*" \
    /Users/elh/adventures/compendium/dm/compendium/5e/ \
    /Users/elh/adventures/campaign-notes/compendium/5e

rsync -vi -rlD -O --ignore-existing --delete --stats \
    --include "*/" \
    --exclude="*.md" \
    --include="*" \
    /Users/elh/adventures/compendium/dm/compendium/5e/ \
    /Users/elh/adventures/campaign-notes/compendium/5e

In both cases, I'm asking rsync to delete files that have been deleted (I'll correct any referencing notes later, but it cleans up duplicates or moved files that I didn't otherwise catch).

• --stats and -vi tell me information about what is happening. If I have a lot of changes, I'll throw -n in there to dry-run first.
• Both use -rlD, which is semi-standard: -r is recursive, -l tells it to preserve symlinks, and -D tells it to leave devices and specials alone (which doesn't apply here, but ... habits)
• Both also use -O to avoid fussing with times on directories.
• Notes are compared with -c, which tells rsync to compare files using a checksum instead of using the timestamp or file size, as generated content will always have a newer timestamp.
• Images use --ignore-existing, so that new images are added, moved/deleted images are deleted, and images are otherwise left alone.

That's the gist, assuming I did everything right. ;)

[mclearc]

I've also been running 5etools via bash script. The following will let me build a compendium, and I can add a -c flag to copy it to my vault and a -t flag to trash the compendium. Of course you can add any other flags you like. A more complex version of the below that I use allows for switching builds between a full compendium and homebrew stuff.

Otherwise the rsync command is pretty similar to @ebullient (explained really clearly above). And as I found out the hard way, don't use cp or mv or you'll send your obsidian vault into overdrive resending eveything.

#!/bin/bash

rm -rf "/5e-cli-compendium/DnD-cli-out" &&
    java -jar /bin/ttrpg-convert-cli/target/ttrpg-convert-cli-299-SNAPSHOT-runner.jar --index \
         -c '/5e-cli-compendium/config.json' \
         -o '/5e-cli-compendium/DnD-cli-out' \
         '/5etools-mirror.nosync/data' \
         '/5etools-homebrew.nosync/creature/Kobold Press; Creature Codex.json' \
         '/5etools-homebrew.nosync/creature/Kobold Press; Tome of Beasts.json' 
        
while getopts 'ct' OPTION; do
    case "$OPTION" in
        c)
            echo "copy (rsync) compendium files" && \
                rsync -vi -rlcD -O --delete --stats \
                      "/5e-cli-compendium/DnD-cli-out/5e-compendium" \
                      "/ObsidianVault/Rules & Options/"
            ;;
        t)
            echo "trash compendium files" && \
                trash "/5e-cli-compendium/DnD-cli-out"
            ;;
        ?)
        echo "script usage: $(basename \$0) [-c] [-t]" >&2
        exit 1
        ;;
    esac
done

[jboehm77]

I know this is old, but wanted to contribute what I ended up using for my setup, as it is relatively user-friendly, which may be ideal for novices like myself:

FreeFileSync is a free software for copying folders and files that has both a checksum option when doing file comparisons as well as a mirror option when performing the sync. It breaks the operation into the distinct compare and synchronize steps so you can have a bird's-eye-view of the changes before overwriting or deleting anything. It can also hook into an actual file comparison app like WinMerge to view side-by-side diffs for individual files.

FreeFileSync and WinMerge together have worked well for identifying numerous unexpected changes and even some bugs (that have since been reported and resolved) since I started using the CLI tool, as well as being very seamless for folder synchronization.

[Samuel1698]

I started using this powershell script since every time I wanted to update the CLI output I needed to run several things. This will take care of running git pull on the data and img folders, remove the old content from the Output folder and executing the convert tool. Optionally it will then copy the contents of the Output directly into the root folder of my vault (It will also delete the current CLI output in the vault if ran like this). I have a private github repo with my vault so I can see any changes or deletions that are caused by this tool before committing them.

Here's how my file structure looks like:

├── Documents
    └── dndcampaign
        ├── # Rest of the vault
        ├── script.ps1
        └── 3-Mechanics
            └── CLI        <-- This folder gets deleted if ran with the '-copy' argument
    └── CLI
        ├── 5etools-img
        ├── 5etools-src
        ├── 5etools-templates
        ├── Output       <-- Folder gets wiped and replaced every time the script runs
        ├── dm-sources.json
        └── ttrpg-convert.exe
    

<#
.SYNOPSIS
    Pull latest data, regenerate Markdown, dump all Output contents
    into the vault root.
.PARAMETER Copy
    Move generated files in 'Output' to the campaign folder
#>
param(
    [switch]$Copy
)
# —————————————————————————————————————————
# Paths (all relative to vault root)
# —————————————————————————————————————————
$vaultDir     = $PSScriptRoot                         # …\dndcampaign
$vaultCli     = Join-Path $vaultDir '3-Mechanics\CLI' # …\dndcampaign\3-Mechanics\CLI : Current Output of the CLI Process
$parentDir    = Split-Path $vaultDir -Parent          # …\Documents
$cliDir       = Join-Path $parentDir 'CLI'            # …\Documents\CLI
$dataRepoSrc  = Join-Path $cliDir '5etools-src'       # …\CLI\5etools-src
$dataDir      = Join-Path $dataRepoSrc 'data'         # …\5etools-src\data
$dataRepoImg  = Join-Path $cliDir '5etools-img'       # …\CLI\5etools-img
$output       = 'Output'
$configJson   = 'dm-sources.json'
$outputDir    = Join-Path $cliDir $output             # …\CLI\Output
$exe          = Join-Path $cliDir 'ttrpg-convert.exe'
# —————————————————————————————————————————

# Update data and images repo
Write-Host "- Pulling latest in $dataRepoSrc"
git -C $dataRepoSrc pull

Write-Host "- Pulling latest in $dataRepoImg"
git -C $dataRepoImg pull

# Remove old build
if (Test-Path $outputDir) {
    Write-Host "- Removing old Dir folder..."
    Remove-Item $outputDir -Recurse -Force
}
if ($Copy -and (Test-Path $vaultCli)) {
    Write-Host "- Removing old CLI output from Vault"
    Remove-Item $vaultCli -Recurse -Force
}
# Run the converter
Write-Host "Running converter in $cliDir..."
Push-Location $cliDir
try {
    & $exe --index -o $output -c $configJson $dataDir
} 
finally {
    Pop-Location
}

if ($Copy) {
    # Dump contents of Output into root directory of vault
    Write-Host "`nCopying everything from Output -> Vault"
    # This copies *all* files & subfolders inside Output directly into the vault folder
    robocopy $outputDir $vaultDir /E /NP /NFL /NDL
}
Write-Host "All done."

Ran by opening powershell in the vault directory and running
./update-compendium.ps1 -copy or
./update-compendium.ps1 to generate an Output without copying to the vault