Collections, thinking over what is probable

[sairuk]

This cropped up in the back of my mind while I was thinking over missing systems, and potential expectations people may have of what RetroNAS is going to look like for them and their existing collections so I'm going to throw it here for discussion

I will add some cautionary notes to demonstrate my thinking moving forward for RetroNAS although any of the below could change with more examples.

---

We probably won't be able to accommodate every existing situation. There will be some kind of reasonable expectations on use, i.e conforming to our generic paths for use and then maintaining an existing 'collection' in those paths if you don't want to duplicate what you already have.

This is of course breaking for say cmpro batch mode (which relies on dat metadata for path creation) and/or TOSEC batched structure which needs to be understood by those looking at RetroNAS with the collector mindset.

For example my entire 20y+ approach to organising my stuff would not have been catered for and I would have fully expected (in the interim) I would need some intermediate solution to present stuff to RetroNAS for use, if that makes sense. While my approach was pretty standard with the older crew at the time it was not use focused.

TOSEC (and all systems therein) for example could theretically be catered for because it has a standard structure but one caveat with the TOSEC layout is how fine grained it is, where it splits software across multiple paths based on media and type. generally is not something supported for obscure emulators and/or frontends.

MAME SL also has a standard layout so these could easily be added, again some software lists are split across media types and this will prove a challenge to integrate fully.

In this situation I wrote myself a script that makes a common single directory for the system then symlinked titles based on a pattern match from folders within TOSEC. It is not a perfect solution by any means and complete breaks the 'collecting on RetroNAS' idea but it sufficed the 'use' of the software.

---

Probably plenty of holes in my thinking above, but worth starting the convo

[CamboG]

This has been bothering me since I started playing with RetroNAS. I'm not sure there's a perfect solution.

Symlinks for directories work great if your generic ROMs folder contains files named and organised in a useable format for running on your target systems. However if your collection uses something like TOSEC naming you probably want to rename and organise your ROMs. Slowly moving down a list of 50+ variants of Sonic 2 on a MiSTer whilst you wait for the title to scroll is not a good user experience. But on another system you might want all those variants, or maybe you a would like a share to use on RetroPie that only includes the ROMs that will run your your Pi Zero.

I've been wondering if something like the Hardware Target Game Database/SmokeMonster system might be the answer. At a very high level it would be reasonably simple.

1. Hash every file in the Generic ROMs folder and store these in a database/text file(s) (update this on-demand or scheduled).
2. Use a SMDB/TOSEC style dat file for each target 'fake' share that defines a target directory structure and file names for each hash.
3. A script/tool creates the target directory structure as defined in the dat file and creates symlinks from the source directory where a hash match is found in the hash database
4. Include an option in the DAT for replicating current behaviour of creating directory symlinks based on pathnames. (make this the default DAT file)

Pros

• Could also be used in a copy mode to create an SD card with a romset targeted to specific ODEs/flashcarts/emulators etc
• ROM set DAT files curated for target systems can be easily shared online/included with RetroNAS
• Could potentially create target directories that incorporate ROMs, saves, bios, firmware, VHDs etc all at a root level (would require the creation of a standard RetroNAS 'Generic Saves Folder', 'Generic BIOS Folder' etc)
• Allows for SMDB style 'Top 100' lists where files can appear multiple times in the target directory structure
• You can organise your 'real' ROM collection however you like (within reason)
• Give best possible user experience for each target system

Cons

• Hashing and comparing takes time (possibly prohibitively long on a Pi)
• Zip files! You can't symlink into a zip! Could see similar issues with CHD and different MAME sets
• Someone has to make and maintain the initial DAT files
• Introduces ANOTHER hash based ROM organisation system
• Difficult for end users to customise and tweak exactly to their liking
• Not real time when adding ROMs since it relies on re-scanning source and hashing new/changed files and then running the script to generate the symlink of the new ROMs to the target directory
• Overly complex?
• Difficult for most users to understand?
• Quite a lot of work and maybe not in the spirit of a project that is more about gluing existing things together rather than reinventing the wheel!

I'm torn as to if it's a great idea or a terrible one!

[sairuk]

Yeh its a big one I like the idea of some kind of romimport functionality being incorporated and easy for people but curation thereof is well out of scope for RetroNAS.

I am going to add SabreTools support at some point so perhaps we or someone can build some kind of frontend around basic functionality that could cover Sabre, Smokes and my own internal tools mapping to RetroNAS folders in a basic UI.

I have no idea how selective you can be with Sabre, I ended up writing my own tool (although its not fast/complete enough to release) but perhaps we could have a frontend for dats support with a focus on importing into the RetroNAS folder structure that let you save profiles for use with these existing tools, and it can be selectively run on a timer to manage the dirs

I used to do this timed scanning with another tool (now abandoned) i wrote, and I could always initiate a manual scan if i needed.

[ZenoArrow]

Hashing and comparing takes time (possibly prohibitively long on a Pi)

There are ways around this. For example, could set up a look up table for hashes and files, so that you don't need to compute the hash on the fly, you're just looking up against known values, which is something a Pi is more than capable of doing fast. Also, you could add in alternative file hashes, so for example if a file has been compressed with a known algorithm, calculate the hash of the compressed file and store that alongside the uncompressed hash in the look up table, so that if the file is compressed it can still be identified (this might not be necessary, just thinking of options).

[sairuk]

I have been giving this some thought recently and this kind of approach is well outside the scope of the RetroNAS tool, we could certainly look at integrating something if it was written much like smoke or sabre and the existing data files out there. I can not see a future where RetroNAS has an 'official set' tho.

The scene has long ago solved this hashing records problem and the data exists in multiple consumable formats (dats, listxml, smoke) so far as RetroNAS goes I think its safe to drop this out of scope, we can run a directory structure that is standard to RetroNAS and can support something reasonable in the way of symlinks for systems with standards of their own providing they make sense to us.

I am thinking about a web frontend for Sabre in the back of my mind that will allow some limited functionality perhaps rebuilding system dats (mame sl, smdb etc) into our directories? It would be pretty rudimentary in the first instance, I don't have a feel for how far that would go either.

I think the community will have to fill in their blanks, we cannot accommodate every situation and if/when it makes sense we can incorporate community tools back into RetroNAS.

SabreTools cli tool is now in RetroNAS so theoretically you could ssh in and start rebuilding into our dirs based on any number of system dats already available from the scene so long as Sabre support it.

[CamboG]

For me one of the most powerful ideas in what makes RetroNAS more than 'just a samba share' is the use of symlinks for the system specific shares. Being able to store ROMs centrally and only once but use them in different ways is a killer feature.

The more I think about it I'm starting to wonder if the current use of Hardware Target Database is backwards! I'm not sure it (or any other organisation add-on) should be modifying the contents of /roms.
What if the SMDB script behaviour worked in the same way as the system specific share setup scripts and was configured to create a 'virtual' output in /SMDB?

The only change required in RetroNAS itself would be the ability to override the default symlink creation for creating system specific shares. The default behaviour shouldn't change but the ability for end users to modify the mapping would allow much more advanced ROM organisation without requiring new tools to be developed.

Imagine my /roms has been organised using TOSEC DATs but I want to use SMDB on my MiSTer. The SMDB script would create linked files in /SMDB and then modify the MiSTer share config and rerun the RetroNAS MiSTer setup which would change the MiSTer share symlinks.

You'd end up with something like this

Original - /roms/sega/genesis/Sonic the Hedgehog 2 Rev 1 (1992-11-21)(Sega).bin
SMDB - /SMDB/Mega EverDrive/1 US - Q-Z/Sonic The Hedgehog 2 (World) (Rev A).md
MiSTer - /mister/games/genesis/1 US - Q-Z/Sonic The Hedgehog 2 (World) (Rev A).md

TD;DR - Should the symlinks that RetroNAS creates for shares be user configurable/remappable to allow for more esoteric ROM organisation techniques/tools?

[sairuk]

Ah yeh we've spoken about file level symlinks as well, it is something I specifically used to do to combine TOSEC (and others) systems into one directory based on pattern matches for use in emulators/frontends that didn't support multiple paths. A dat scanner that symlinked sets to save space was something i'd intended to write over the last few years but I haven't found the motivation/time.

While yes the generic rom dir->system symlinks are a killer feature of RetroNAS, we still need the tooling. MiSTer specifically on the other hand is only a small portion of the RetronNAS feature set and only needs samba+SMDB, where SMDB is the accepted tooling for the space we have both of these right now.

I had a look and the tooling for SMDB doesn't support file level symlinks (nor symlinks at all) so we'd either need to write our own solution OR get smokes tooling updated to add support (unlikely I imagine)

Still this is good discussion as it allows those who may be capable of writing the tooling to consider all options, so thanks for the continued input