Orgzly Revived WebDav implementation vs Directory repository in Davx5 webdav mount?

[sethidden]

I noticed that Davx5 supports webdav https://manual.davx5.com/webdav_mounts.html

Material Files similarly allows you to connect to a remote SFTP, Samba or WebDAV mount

That left me wondering - is keeping the WebDAV sync repository implementation within Orgzly Revived still necessary? I can just set up webdav within davx5 - that will create a new filesystem within Android. I can then use that webdav mount with Orgzly Revived using the "local" repository, rather than the webdav one.

This is the approach used by KeePassDX - https://github.com/Kunzisoft/KeePassDX/wiki/File-Manager-and-Sync, where it doesn't implement any file syncing protocol, but instead expects an external program (Syncthing, Material Files) to provide file access.

I don't really have any serious thoughts on the matter, just wanted to start the discussion

[amberin]

Sounds great - I hope someone can try this out. It might require some tweaks to ContentRepo to parse book names correctly, but that shouldn't be hard once we see the resulting URIs.

[amberin]

I will try this out myself, since #312 apparently broke this. I suspect there are a few different ways to make it work, although none of them are pretty. I have a feeling that there is no pretty way of achieving this, because we are clearly working against Android's design here. I'm thinking of this disclaimer in the DocumentFile docs (emphasis is mine):

• A single document may appear as the child of multiple directories, so it doesn't inherently know who its parent is. That is, documents don't have a strong notion of path. You can easily traverse a tree of documents from parent to child, but not from child to parent.
• Each document has a unique identifier within that provider. This identifier is an opaque implementation detail of the provider, and as such it must not be parsed.

[amberin]

I've found a way to build a "repo relative path" from any content:// file URI. It works, but it's expensive and convoluted. Remains to be seen how slow it will be.

[amberin]

I've noticed a problem with this kind of setup, unrelated to subfolders (it exists in 1.8.26).

When syncing local changes, everything looks good, but upon the next sync, the book is loaded from the repository, instead of showing the "no changes" status. It is caused by the repo file's modification time being different right after writing and when retrieving all files during the next sync.

This means that two syncs are required before Orgzly considers the book to not have any changes. More importantly, if I make a change in Orgzly, sync, and then make another change in Orgzly, upon the next sync I get a "both local and remote notebook have been modified" error for the book in question... Which obviously renders this setup pretty useless, in its current state.

Are you seeing the same behaviour, @sethidden? It seems like something that should cause syncing trouble pretty quickly.

The issue does not appear when using a local directory repo (folder on the phone) or a regular WebDAV repo with the same servers. So it seems to be related to the interactions between Orgzly, Android, DAVx5, and the WebDAV server. We use the same DocumentFile method for retrieving the file's modification time in both occasions, and the same DocumentFile URI, so something happens to the "file" (DocumentFile) in the meantime.

I tried two different WebDAV servers, one local minimal implementation (dave) and one cloud-based, and both behave the same way.

[sethidden]

I haven't come across an issue where my changes were lost. I do get the "both local and remote notebook have been modified errors" frequently but then I just force upload to remote. I use Orgzly purely on my phone lately rsther than with my phone and Emacs as in the past, so I can get away with that. I'm using mailbox.org's WebDAV server. I'm nit sure whats the underlying implementation.

…

On September 7, 2024 2:08:52 PM GMT+02:00, Victor Andreasson ***@***.***> wrote: I've noticed a problem with this kind of setup, unrelated to subfolders (it exists in 1.8.26). When syncing local changes, everything looks good, but upon the next sync, the book is loaded from the repository, instead of showing the "no changes" status. It is caused by the repo file's modification time being different right after writing and when retrieving all files during the next sync. This means that two syncs are required before Orgzly considers the book to not have any changes. More importantly, if I make a change in Orgzly, sync, and then make another change in Orgzly, upon the next sync I get a "both local and remote notebook have been modified" error for the book in question... Which obviously renders this setup pretty useless, in its current state. Are you seeing the same behaviour, @sethidden? It seems like something that should cause syncing trouble pretty quickly. The issue does not appear when using a local directory repo (folder on the phone) or a regular WebDAV repo with the same servers. So it seems to be related to the interactions between Orgzly, Android, DAVx5, and the WebDAV server. We use the same DocumentFile method for retrieving the file's modification time in both occasions, and the same DocumentFile URI, so something happens to the file in the meantime. I tried two different WebDAV servers, one local minimal implementation ([dave](https://github.com/micromata/dave)) and one cloud-based, and both behave the same way. -- Reply to this email directly or view it on GitHub: #304 (comment) You are receiving this because you were mentioned. Message ID: ***@***.***>

[amberin]

I think I found the cause. Both WebDAV servers that I used seem to store modification times in seconds, whereas Android stores them in milliseconds. On the WebDAV server, the time is rounded up to the next second, and this is why we get a diff later, because the WebDAV server reports a different timestamp.

If I'm right, it should be easy to solve this by always rounding up mtime (or revision, or both) to the nearest second - at least when making this comparison between last seen file revision and the latest one in the repo.

[amberin]

Nope, it wasn't that simple. Rounding up when comparing is necessary, but the timestamps can differ by quite a lot.

It seems that the modification time reported by the DAVx5 ContentProvider is not updated when we write the file, but only once the changes have been fully synced to the server, which can take quite a while. So whenever we store a file to the repo, we store its previous modification time as if it was the new one.

On top of that, when we fetch books which were recently updated, we first see a file mtime with millisecond precision, but a while later, the same action returns a timestamp rounded up to the next second. So the mtime changes two times after we have finished writing our file, making it useless as a revision identifier.

[amberin]

I'm afraid it is now pretty hard for me to justify putting effort into making this sync setup work well, since anyone who is actually syncing in two directions will have no way of knowing whether there was a real conflict or not.

It's too bad, in a way, because I believe I now have a better way of identifying book names in content:// repo subfolders, prompted by this use case.

[amberin]

I will suggest adding a "second-precision" mtime comparison whenever the millisecond-precision mtime comparison fails. This way we can at least handle the situation where the underlying ContentProvider stores mtime in seconds, not milliseconds.

There are other problems in this domain, but I think #536 will lead to some changes soon which should at least improve things.

[amberin]

I looked into such a second-precision mtime comparison, but my tests with a free public Nextcloud share (thegood.cloud) were pretty discouraging. I expected that the mtime reported by the WebDAV server would be rounded up to the next second, but what I found was that it got rounded down, sometimes by more than two seconds compared to Orgzly's own timestamp of when it wrote the file. Such a big discrepancy cannot be explained by latency anywhere - Orgzly's sync process takes about 500 milliseconds from start to finish with this build. So something weird is definitely going on at the WebDAV server side -- it's not just rounding up or down the timestamp from when Orgzly writes (or deletes or creates) the file.

I won't look more into this right now. Let's hope that the changes from #381 at least makes this setup more viable.

[alensiljak]

Perhaps not directly related to the question but potentially useful alternative. I've moved away from using WebDAV exactly because of the issues mentioned. I find explicit synchronization using rclone much more reliable. RClone supports many cloud providers and that way I can keep 2FA on the cloud storage account whereas with WebDAV I could not (due to the provider). I use it mainly to transfer binary files or files that are rarely modified at multiple locations. For notes and project files (*.org), I find git to be the best option.
I still believe that editors, be it Orgzly or KeePassDX, should not really integrate synchronization. That should be handled separately, simply because there are so many potential destinations. It would be better to provide a link (intent) which can be called to perform the synchronization. And the synchronization itself should be in a specialized app that synchronizes any type of file between the device and the remote provider.