System tiddlers content hash, support updateas, patches

[AnthonyMuscio]

Folks,

I was wondering if we could build a System tiddler content hash for tiddlers in the core?. This would be stored in a hidden field on each system tiddler, or regular ones by design. I believe it serves a few identified purposes below but such a facility could be used in other applications if made available as widgets.

Basically as a new release is issued an algoritium runs against each system tiddler which is analysed and generates a hash, ideally using a method to keep the hash size as small as possible (eg use a base 60 encoding) whilst giving those interested the ability to test if the system tiddler has changed. Yes this is in addition to shadows vs tiddlers. Basically a good hash will allow you to detect if even one character in the tiddler changes. Ideally this would apply to the whole tiddler but text would be enough.

Now if the hashing algorithm is made available to us, we can test a given system tiddler for changes, if the stored (published) hash does not match the computed hash. This could be used for the upgrade utility to quarantine edited system tiddlers that have being modified, and Plugins forced to modify a system tiddler, could detect if an upgrade has changed an underlying system tiddler it wants to modify. Ideally this would trigger a diff view one could choose to update or maintain the changes selectively.

Most of all such a facility could give rise to the ability to distribute core hacks or patches which can detect if and when a plugin or upgrade has changed a system or shadow tiddler. This will be of great benefit to designers and developers who want to test and share possible core changes with a patch.

Plugin authors could detect changes to their own tiddlers and include handling as desired. Some beautiful Agile and component based development solutions could be built on this alone, not to mention the ability to use this to manage data plugins and as a de facto packable delete, retain the hash but delete the content.

Your thoughts?
Tony

[AnthonyMuscio]

Post script:

Sure a plugin designed to do this could be made but it would change all shadows, that's why it should be slipped into the core or a special edition.

Importing tiddlers whose has hash not changed could be discarded and it made clearer to the importer which tiddlers actually changed on import/overwrite.

Perhaps modified date needs to be left out of the hash.

Tony

[pmario]

The simple check, if a system tiddler is overwritten is: [is[tiddler]is[shadow]sort[title]]. This is part of the $:/ControlPanel : Info : Basics tab. ... BUT ... This info doesn't really show if the tiddler content has been changed. It could have been opened and saved without changing something.

---

I was wondering if we could build a System tiddler content hash for tiddlers in the core?. This would be stored in a hidden field on each system tiddler, or regular ones by design

That doesn't work. ... If the hash is part of the system tiddler I can create a plugin, that overwrites a core tiddler and add the hash of the new tiddler to that field. ... Since the hash-field can not be part of the hash you have no chance to detect, that I did create a new system tiddler, that looks like core, because the hash is valid. ... We can only trust a hash, if it comes from a "trusted source" outside the wiki.

The only way that works is, that the wiki build process will create the hashes and publish them separately and versioned. So if a user or a plugin author wants to check the code has been modified, they would calculate the hash of the wiki-tiddler and compare it with the hash of the original-tiddler published at GitHub.

---

If we use a sha256 hash, which should be secure for now, it has 32 characters. Changing the encoding will make validation more complicated.

[AnthonyMuscio]

Mario,

The purpose of my proposal is not to secure tiddlers against edit, I accept when a shadow becomes a tiddler, the hash can be updated etc... The intention is really just to test if an actual change has occurred in the tiddler through a simple edit. The hash would be used more like a CRC - cyclic redundancy check.

If for example the update process is about to replace a core tiddler, if it discovers it has a tiddler (overwritten shadow) and it checks a newly minted hash with the hash field value (put there in the last release) it can determine if a substantive edit has occurred. The tiddler can be moved into a quarantine JSON of tiddlers, and after the update it can report all tiddlers so quarantined and replaced for manual review (with a dif and compare tool). Since the new core tiddlers that have changed now have a different hash, a plugin could determine that the core tiddler has changed underneath it - eg may need a repatch and could prompt the user to repatch/edit the now different core tiddler.

[AnthonyMuscio]

Can I ask what character set does sha256 use? 0-9-A-Z?

[pmario]

... Yes this is in addition to shadows vs tiddlers. Basically a good hash will allow you to detect if even one character in the tiddler changes. Ideally this would apply to the whole tiddler but text would be enough.

IMO "text" would not be enough. TW allows us to use a type-field: application/javascript ... which makes a tiddler an executable. So without changing the text we can completely change the behaviour of a tiddler. So the hash would need to include everything.

[AnthonyMuscio]

Agreed, but I am not trying to protect tiddlers, just detect when they are different, we can decide which if any fields are included in the hash, thus if edited can be detected, eg tags or as you suggest the type field.

[pmario]

For plugins, it would be enough to have 1 hash for the whole plugin including all tiddlers.

[AnthonyMuscio]

I don't agree, what if I only change one tiddler in a plugin. In fact it may be a config tiddler. I would like a way to detect which tiddler was edited.

[pmario]

If you change a config tiddler, that comes with a plugin and you change the tiddler, doesn't change the plugin itself. Tiddlers in a plugins are "extracted" at startup. So you can easily compare tiddlers that have been changed. see:

• [is[tiddler]is[shadow]sort[title]]
• You can test that at tiddlywiki.com
• Open https://tiddlywiki.com/#%24%3A%2FDefaultTiddlers

see:

[pmario]

If you can trust the hash of the plugin, you can safely compare the tiddler content using default tools

[pmario]

... We can only trust a hash, if it comes from a "trusted source" outside the wiki.

This gives us a new problem. .. How can we verify, that nobody did mess with the hashes? ... Compress the hashes into 1 file and sign it with PGP.

Similar to: https://wiki.debian.org/Creating%20signed%20GitHub%20releases

[AnthonyMuscio]

Sorry, what I mean here, and may have misled you. Is lets generate the hashes when publishing a new version of the wiki eg; 5.2.1 these changes will remain the same indefinitely until a future version makes a change and the hash changes. My trust here not that it has not being maliciously tampered with but the hash represents the published version.

[AnthonyMuscio]

A record of hashes at version publishing could be held in a separate plugin and lookups done but this is more complex, although the core need not be changed.

[AnthonyMuscio]

Sorry, I am still working through the whole story, but if you can see my reasons, perhaps we can find a solution.

So my idea is that perhaps the vast majority of core tiddlers do not change between versions, if a plugin applied a patch to a system tiddler with a particular hash then if that hash does not change then the core tiddler has not changed and the same patch can be retained or reapplied. If the core hash has being changed then all bets are off and the patch cant be reapplied without intervention.

In some ways this would allow every tiddler in the core to have its own change flag, even a hash for each version giving finer grained control and the ability to write patches to a specific "version" or "hash" of a core tiddler.

[AnthonyMuscio]

Mario,

Your last response makes me wonder if you see what I want because how will you do it without the core?

I see that the tiddler save mechanism and visibility would be part of the solution, but first we need a hash algorithm we can run against any tiddler such that a single character change will result in a different hash. Now we can apply these to every tiddler but they will change from shadows to tiddlers, alternatively we could store them separately in a data tiddler with the tiddler title as key, however I would think the published core hashes need to be in place before any plugin or patch can make use of it. Each empty that is released we could generate a hash database?

Perhaps we can stop discussion ping pong and have a one on one discussion?. Can you find a time that may work with my UTC +10 and I am sure we can simplify the discussion?

Regards
Tony

[pmario]

I see that the tiddler save mechanism and visibility would be part of the solution, but first we need a hash algorithm we can run against any tiddler such that a single character change will result in a different hash.

Try this at tiddlywiki.com in the F12 browser console.

window.sjcl.hash.sha256.hash("asdfsaaf").map((i) =>{return (i < 0 ? 0xFFFFFFFF + i + 1 : i).toString(16).toUpperCase();}).join("-")

[AnthonyMuscio]

If this is hashing asdfsaaf into "4CF02349-315A72CE-5CB96C71-B4757A46-43BBC7AF-ABF85604-3CE3611C-EEB37AB5" we may as well retain a copy of the content, rather than a hash, or am I missing something?

[pmario]

I did send you a PM with my Skype handle.

[AnthonyMuscio]

Mario, I will try and contact you at an appropriate hour in the next few days.