Syncing your current saves folders, wallpapers, configs, into the root of the MiSTer share

[sairuk]

@birdybro had this cool idea over here retronas/retronas#50 (comment) about syncing additional parts of the MiSTer config across to the RetroNAS so its accessible.

I'd like to flesh this out a bit more here, There are many people with more MiSTer experience here than me, so there may be some fundamental info I'm currently missing to facilitate this.

Functionally I am thinking

• sshfs automounted remote share of the MiSTer filesystem
• systemd path units to monitor the share from RetroNAS side in a pull config
• rsync on trigger to sync data between the RetroNAS

On the RetroNAS I'd expect the filesystem layout would look similar to

/data/retronas/mister/games
/data/retronas/mister/saves
/data/retronas/mister/savestates
/data/retronas/mister/<etc>

In ansible I'd be looking to do something like the below snippet to manage paths on RetroNAS. It should ensure it retains the conformance as per pull retronas/retronas#50

https://github.com/danmons/retronas/blob/fix-mister-paths-alt/ansible/install_mister_cifs.yml

There is a new branch you can pull, checkout and run with the new startup scripts to create the resulting layout in the mister share. I'd suggest you test on a non-prod deploy of RetroNAS

Open for comment, systemd path files may not be the best idea for monitoring/sync. I figured this concept should be captured somewhere other than a pull request

[citizen-why]

So I'm certainly not an expert in this space - but am excited for this feature.

I've been sharing save files via a standard NAS for sometime now - but have done it simply by making the "saves" folder in mister a mounted folder just like "games". In this mode I've been able to use the same save files on multiple MISTer devices in my house just by booting them up (since they automount a common save folder from the NAS).

I was trying to follow how what issues the following functionality was solving for:

• sshfs automounted remote share of the MiSTer filesystem
  

• systemd path units to monitor the share from RetroNAS side in a pull config
  

• rsync on trigger to sync data between the RetroNAS
  

As I said I'm no expert here so feel free to ignore the question as pure ignorance.

[sairuk]

Thanks @citizen-why I'd really like more details on your setup, we have recently refactored the mister playbook so it has the ability to create saves/savestates folders and other top level folders in the mister share but I have not had the chance to test this. We have just disabled the creation of these other dirs for now.

Are you saying this is functionality you are currently running of a share and it works OK? I am currently looking to get around to testing it when I am able but if you are already doing so and it is pretty low resistance to test I'll fit it in at some point.

The proposals I made would only come into affect if a simple automounted fileshare wasn't to suffice in all cases. I probably could have been clearer to that end, sorry about that.

sshfs is a filesystem client that can be installed to the rpi (linux) and used to remote mount a hosts filesystem over ssh with the end result being kind of like samba or nfs, We could automount that at mister boot and perform a sync based on once of the other trigger method (systemd path, or a watchdog trigger to rsync)

I currently do an sshfs automount here at home with mister and it works well enough for access like a normal share.

[sairuk]

i am currently looking at updates for rom, saves etc dir layouts generic and mister support that should be able to translate to other system support long term

[citizen-why]

Happy to share how I am setup today. I do warn you though - not an expert, so there may be limitations that I'm not aware of. All I know is that this is working for me without issue.

So at its heart - I took the philosophy that I want as little as possible store on the MISTer SD cards and wanted to have as much possible shared commonly on the NAS. So here is the filestructure of my NAS

You'll see I actually have it setup such that the MISTers not only share the "games" directory and "saves" but also share all cores, and cheats, filter files, fonts, etc. The expansiveness of my setup is probably not ideal for all, but just sharing as an example so you know how the "saves" implementation works.

With this structure in the root of my remote mount point... I can just use a fairly typical cifs_mount.ini (screen shot below)...

[sairuk]

That is brilliant, thanks @citizen-why, this certainly goes towards answering some question in the back of my mind. and certainly saves me some work writing a sync script.

At this point I am still assuming the saves dir reflects directory names as per the games folder. Not something I have looked into yet.

[citizen-why]

I think you are right about the saves dir - here is what mine looks like (now the world can see which consoles I use the most).

I imagine if you want you could treat them similar to the games directory and use symlinks to point back to multi-purpose directories that can be used generically across other platforms. In the individual console directories it looks like the save files are all in the same console directory named with a common name as the rom file that generated them.

For posterity - it looks like the savesstates directory follows that same standard (though not sure it worth doing the same thing for as the saves directory) - here is mine as well

For posterity I'

[sairuk]

Great thanks for confirming that, this is how I setup the playbook so it is nice to see if confirmed.

[sairuk]

I've updated some of playbooks today with additional logic allowing a top level dir to be made as either a single dir or a dir with system symlinks and I've migrated the romdir playbook to use the same base systems list as we did for mister support with the goal to have a single source of truth for system level mapping.

If someone wants to have a go and migrating the ps3netsrv or ops2l to use the retronas_systems playbook please feel free otherwise i'll get to it at some point.

I am going to do some more testing on these new playbooks before I enable support by default, just a question of how far we go, as per @citizen-why setup we can arguably do everything on RetroNAS but do we want to? Open for discussion.

[citizen-why]

@sairuk any chance you could add the symlink for the saves and savestate folders in the mister directory root - would love to have it for testing purposes

[sairuk]

I wont be enabling it for everyone yet but if you want to test it you can do so by editing this line and setting enabled: "yes"

https://github.com/danmons/retronas/blob/1ea62b4413084eacd31f19fe38d7a90cdfc4330f/ansible/install_mister_cifs.yml#L9

then run retronas with the -g switch which will turn off the automated git updates e.g. retronas -g

Finally reinstall the Samba for MiSTer option to trigger creation of the top level saves dir.

[citizen-why]

Thanks for this - just a quick update, I've had this working for the last 3 days and its working like a charm. Will report back with any issues.

[citizen-why]

Another quick update - been running for nearly 4 weeks now and managing save files across multiple cores. No issues at all - runs flawlessly.

[sairuk]

Alright, sounds like we should probably think about enabling it as the default

[citizen-why]

So I was thinking through what framework to use on which directories should be included in retronas and wanted to propose the following criteria for determining which to INCLUDE on retronas:
1.) Content that is owned / curated by the user (CURATED)
2.) Content that is specific / created by the user (CREATED)

I propose the following criteria on what to EXCLUDE on retronas:
1.) Content is regularly updated / mantained by the community (COMMUNITY_MANTAINED)
2.) Content that is inherently specific to a single device (DEVICE_SPECIFIC)
3.) Content that is required at run time or when network storage is not available (RUNTIME)

With that in mind I'd suggest INCLUDING the following:

• BIOS (CURATED)
• games (CURATED)
• saves (CREATED)
• savestates (CREATED)
• MIDI (CURATED)
• MP3 (CURATED)
• Presets (CREATED)
• wallpapers (CURATED)

And EXCLUDING the following:

• _Computer (COMMUNITY_MAINTAINED)
• _Console(COMMUNITY_MAINTAINED)
• _LLAPI (COMMUNITY_MAINTAINED)
• _Other (COMMUNITY_MAINTAINED)
• _Utility (COMMUNITY_MAINTAINED)
• inputs (DEVICE_SPECIFIC)
• Scipts (RUNTIME)
• config (RUNTIME)
• linux (RUNTIME)

Items that could go EITHER WAY (probably should default to excluding

• Shadow_Masks (CURATED) - this can be argued either way
• Gamma (CURATED) - this can be argued either way
• cheat (CURATED) - this can be argued either way
• Filters (CURATED) - this can be argued either way
• Filters_Audio (CURATED) - this can be argued either way
• font (CURATED) - this can be argued either way

Just a framework as a start

[birdybro]

Personally, almost all of the folders that are downloaded by the update script should be included so people don't have to rerun the update script if they have multiple MiSTer's in the same network. I dunno.

So, in my opinion, the folders that should be on retroNAS by default with a toggle, and (hopefully) will get updated by the downloader.sh script, should be:

• _Arcade
• _Computer
• _Console
• _Other
• _Utility
• BIOS
• Cheats
• Filters
• Filters_Audio
• font
• games
• Gamma
• MIDI
• MP3
• Presets
• Shadow_Masks

The folders that are in the optional category should be:

• saves
• savestates
• wallpapers

The folders that should be absolutely excluded:

• config (this also contains inputs, so I removed that, as it's redundant)
• linux
• Scripts

My reasoning is... multiple MiSTer's in one network, maybe someone has their kidsafe one where the kids won't ruin their saves, or see their scantily clad big anime titties wallpaper on shuffle, or ruin their best finish savestate, etc... The default on at the top are pretty much everything that the updater touches and should be assumed default behavior as a result. This will reduce the time the update scripts that are smart, take to run. The ones excluded are for the same reasons you mentioned, runtime. However scripts in the scripts folder aren't necessarily at runtime, they could be sent to default position in the top category. The default scripts that run at runtime should be in either the linux folder or in system folders up a couple levels.

I'd appreciate @theypsilon's input on this as he's very familiar with the user's flow with all this.

[Paradox]

I only have one mister, at the moment, but, I agree, if I end up purchasing another one, or, more than one, in the future, I'd want one run of the update script to update all of them, instead of having to update each one, separately.

[Paradox]

The beauty of the Mister is that you can still have some content on the NAS and some on a USB drive, for example, and the system will integrate the two sources (with the USB taking precedence over both the fat and cifs directory) so, you can still control how the setup is utilized, right now, I have all of my ROMs on a USB HDD, attached to the MiSTer, except for CD based, such as PSX, Sega CD, Turbo Grafix CD, on the NAS, and it is seamless.

[Paradox]

I'd also suggest that maybe you set it up with all of the directories, except those @birdybro mentions as definitely exclude, and use the local_dir= entry in the cifs script to control specifically which directories are actually used, setting it up to only include the directories @birdybro mentions, by default, and, allowing the user to edit the line to include, or, exclude, whatever they specifically want, for their system.

and, to make it new user friendly, you could have more than one local_dir= line, with all except one commented out, so, all a user would need to do, is determine which set of directories they want, comment out the default one, and uncomment the one that (closely) matches what they want...

[Paradox]

also, note that the special_directories= entry already prevents directories, such as config and linux being mounted, so, you could leave local_dir=* and use this line to prevent it mounting particular directories.