Updating modification times in --sync can cause problems with editors

[td-anne]

The VSCode plugin for Jupytext (https://github.com/caenrigen/vscode-jupytext-sync) is having a problem with file modification times: caenrigen/vscode-jupytext-sync#12

Specifically, the problem is that when a paired file is open in the editor and is saved (having been changed), the plugin runs jupytext --sync. This updates the other version of that file, then changes the timestamp on the file that was just saved. The editor then detects that the version of the file on disk is newer than the time the open editor saved it. If the user has made some change by this point, the editor sees active changes in the editor, and a file on disk that has changed since it was written out - saving again could cause data loss, so the editor pops up an error that the user has to deal with.

(In VSCode, if the user has not changed the contents of the open editor, the editor attempts to reload the file from disk, updating the active editor. This is unfortunately not without impact, at least for the notebook editor; it can mess up capturing the output from running cells, and it drops the user out of cell-editing mode.)

I don't have a simple solution for this, and maybe it has to be somehow handled by the editor plugin, but the phenomenon can arise even if one is running the jupytext --sync by hand from the command line.

Not updating the modification time of the original file leaves the timestamps in a misleading state, where the original appears older than the converted file; another --sync run will copy data the other way, possibly clobbering any changes to the original made before the converted file finished being written.

Setting the modification time of the converted file to exactly that the original had when it was checked would seem to resolve these issues, but I'm not sure that's possible on all operating systems/filesystems.

Here is a python file that demonstrates the behaviour (though not, obviously, its interaction with the editor):

import time
from pathlib import Path
from subprocess import run
from tempfile import TemporaryDirectory

PYTHON_CONTENTS = """

print("Hello")

"""


def main():
    with TemporaryDirectory() as tmpdir:
        tmp_path = Path(tmpdir)

        py_path = tmp_path / "jupytext-test.py"
        ipynb_path = tmp_path / "jupytext-test.ipynb"

        py_path.write_text(PYTHON_CONTENTS)

        start_time = py_path.stat().st_mtime

        time.sleep(0.01)
        # Run jupytext to convert the script to a notebook
        run(["jupytext", "--set-formats", "py:percent,ipynb", str(py_path)], check=True)
        post_pair_py_time = py_path.stat().st_mtime
        post_pair_ipynb_time = ipynb_path.stat().st_mtime
        print(
            f"Post-pairing times: py={post_pair_py_time - start_time}, ipynb={post_pair_ipynb_time - start_time}"
        )

        time.sleep(0.01)
        # touch .py file to update its timestamp
        with open(py_path, "a+") as f:
            f.write("\n# Touching the file to update its timestamp\n")

        post_touch_py_time = py_path.stat().st_mtime
        post_touch_ipynb_time = ipynb_path.stat().st_mtime
        print(
            f"Post-touch times: py={post_touch_py_time - start_time}, ipynb={post_touch_ipynb_time - start_time}"
        )

        pre_sync_py_contents = py_path.read_text()
        time.sleep(0.01)
        run(["jupytext", "--sync", str(py_path)], check=True)
        post_sync_py_time = py_path.stat().st_mtime
        post_sync_ipynb_time = ipynb_path.stat().st_mtime
        print(
            f"Post-sync times: py={post_sync_py_time - start_time}, ipynb={post_sync_ipynb_time - start_time}"
        )

        post_sync_py_contents = py_path.read_text()


if __name__ == "__main__":
    main()

[mwouts]

Hi @td-anne , thank you for the detailed report. I can see how this is a problem, and I am open to try setting the timestamp to that of the last modification. We will need throughout testing of this - We could also think of a way to deliver this that is not too impactful e.g. through a new option or through a release candidate. I will try to have a look soon.

[td-anne]

For my application (I just want to edit notebooks in VSCode and have the .py files keep up), an additional option that the plugin could use would be sufficient. If it were implemented as a --keep-original-timestamp/--no-keep-original-timestamp pair, no change should be apparent to users who don't use the option, and the possibility of changing the default if this should prove helpful is open. Possibly three options could be provided - update the original's timestamp to be newer than the converted file, change the converted file's timestamp to match the original, or do not update either timestamp? If it were not clear what the relative benefits of the different options were.

[mwouts]

Hi @td-anne , @caenrigen, I gave a quick look at the past issues and I found this: #790.

In other words jupytext has a --use-source-timestamp option. Could you give it a try in the context of --sync in the VS code extension and let me know if that solves the issue? Thanks!

[td-anne]

Ah, I'm sorry I missed that! Looking at my log messages, though, I get some peculiar entries:

[jupytext] Setting the timestamp of [filename].ipynb equal to that of [filename].ipynb

That is, jupytext appears to be setting the timestamp of the notebook file to a timestamp drawn from the notebook file. This seems pointless at best and a race condition at worst (what if I modified the notebook between when the original timestamp was read and the new timestamp was written? It'll make the notebook look older than it actually is). Is this intended behaviour?

(I've no idea whether this is related to this particular bug, but this particular bug does not appear to have disappeared even using the --use-source-timestamp option.)