Writes to .ipynb file even when it is newer (being actively edited)

[td-anne]

I am working with the .ipynb file (I don't have the paired .py file open and am not editing it) and from time to time the .ipynb file will be rewritten by jupytext. Mostly this results in the .ipynb editing window dropping out of editing the cell I am editing, sometimes scrolling to somewhere radically different in the notebook; sometimes it somehow interrupts the kernel executing the current cell, or at least swallows the cell's output; occasionally I get a message telling me that the .ipynb file is newer than VSCode thinks it should be and I have to manually tell it that it's safe to overwrite it with what I am editing.

I have VSCode autosave on pauses in typing, so I think this happens when VSCode saves the .ipynb file; definitely I see it happen (sometimes) when I manually hit "save". I also have ruff formatting the file on save.

My guess would be that the .ipynb file is being saved, then jupytext runs and updates the .py file; but then the .py file appears to be newer, and jupytext runs again, copying the content of the .py file back into the .ipynb file. File-wise, this is a safe round-trip, so it does not damage the contents. Unfortunately VSCode correctly detects that the .ipynb file has been "modified" and reloads it; this reload affects active editing.

[caenrigen]

Hi Anne,

All of this sounds strange...

I am using ruff as well to format the .ipynb file and I am usually editing only the .ipynb as well. However I don't use the VS Code autosave feature.

I can imagine that something can go wrong when several pieces of code trigger on the same "onSave" event without being aware of each other. Can be specially problematic if both ruff and Jupytext Sync trigger the same "onSave" event again in a circular loop.

I don't know if VS Code extension's API even provides any means to cleanly deal with such a problem.

How easy is it to reproduce the problem that you are describing? Does it happen every few saves or is it a once-a-day type of issue?

Would be good to share as well the specific version of VS Code/ruff/Jupytext Sync you are using.

[td-anne]

On my local machine, where the problem occurs frequently, I am using jupytext 1.16.2, Jupytext Sync 1.1.5, and VSCode 1.102.1. The problem occurs locally on MacOS and remotely on a Linux server.

The problem occurs frequently, but not every time, with both manual and automatic saves, although I think only if the notebook has changed (maybe even only if the notebook has changed in a way that would impact the generated .py file).

I don't actually know that ruff works with files at all - it may be processing the data in memory and irrelevant here. It is the most visible effect of a manual save, though, as it reformats my code. Since the problem also occurs on autosave, and ruff does not format on autosave, the formatting may be a red herring.

Is it possible that Jupytext Sync's changes to the .py file are also triggering an onSave()? I wouldn't have thought so without the .py file even open, but that could explain the behaviour.

Whatever is triggering it, it seems as if jupytext --sync is running twice, once to copy changes to the .py file and then again, writing something to the .ipynb file. Whether this happens could depend on things like the granularity of the timestamps and the time required to finish saving.

Looking at the jupytext output window I have noticed that when I manually save in close proximity to an autosave the messages can become intermixed:

ead paired formats for '[...].ipynb': 'ipynb,py:percent'
Read paired formats for '[...].ipynb': 'ipynb,py:percent'
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Unchanged [...].ipynb
[jupytext] Updating [...].py
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Unchanged [...].ipynb
[jupytext] Updating [...].py

The normal behaviour, only one save action at a time, looks like:

Read paired formats for '[...].ipynb': 'ipynb,py:percent'
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Unchanged [...].ipynb
[jupytext] Updating [...].py

After an incident where the "Failed to save" was triggered I got:

Read paired formats for '[...].ipynb': 'ipynb,py:percent'
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Unchanged [...].ipynb
[jupytext] Updating [...].py
Read paired formats for '[...].ipynb': 'ipynb,py:percent'
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Loading [...].py
[jupytext] Updating [...].ipynb
[jupytext] Unchanged [...].py

Although there are no timestamps on the output so I can't say for sure that these both happened on the same save event, they would produce the UI behaviour I see: the .ipynb file is changed, so the .py file is updated; then for some reason the .py file is thought to have changed and the .ipynb file is changed to match it. In this case I was actively typing, so the in-memory .ipynb file is different from the one on disk, and VSCode complained about the on-disk version being newer. If I hadn't changed it before the "updated" .ipynb appeared on disk, VSCode would have reloading it, kicking me out of editing mode and possibly seeking to somewhere random in the notebook.

[caenrigen]

Thank you for the detailed input.

I am using jupytext 1.16.2

Could you upgrade to the latest version? there is a bug in order versions related to timestamps that might be indirectly related

Is it possible that Jupytext Sync's changes to the .py file are also triggering an onSave()?

In principle this should not happen afaik. jupytext --sync runs only on the file being saved

---

Thank you for the logs that is helpful!

Looking at the jupytext output window I have noticed that when I manually save in close proximity to an autosave the messages can become intermixed

This might be the main culprit. The Jupytext Sync extensions is coded in asynchronous typescript (javascript). This has the side effect that more than one jupytext --sync can run in parallel. From your logs, this seems to be exactly what is happening in your logs.

A potential solution could be some kind of global "locking mechanism" to guarantee that there is only one and only one jupytext --sync command running at a time for each file being synced. Though locking mechanism can be sometimes tricky to implement.

I suggest for now to avoid mixing autosave and manual save, and rely on a manual save instead, annoying I know...

However, I suspect that there is something more happening in there, either related to the auto formatter or something else.

Do you happen to be pressing cmd+s several times in a raw when saving a file? I do it sometimes unconsciously and it might trigger an equivalent situation of several jupytext --sync commands running in parallel.

[td-anne]

I did try to trigger the problem by hitting ctrl-s repeatedly, but with no real effect. I think the problem is easier to trigger on remote sessions, which fits with the idea that it's concurrency-related - saving and running jupytext sync both take longer there.

I will note that the logs from when the problem actually happened do not seem to show two sessions intermixed.

The problem persists with the new jupytext. In this case I definitely did not save manually, the autosave fired during a brief pause in typing. (Reloading the .ipynb booted me out of edit mode and I found myself typing into the keyboard shortcuts.) Another log sequence (sorry I wish they had timestamps):

Read paired formats for '[...].ipynb': 'ipynb,py:percent'
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Unchanged [...].ipynb
[jupytext] Updating [...].py
Read paired formats for '[...]': 'ipynb,py:percent'
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Loading [...].py
[jupytext] Updating [...].ipynb
[jupytext] Updating the timestamp of [...].py
Read paired formats for '[...].ipynb': 'ipynb,py:percent'
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Unchanged [...].ipynb
[jupytext] Updating [...].py

I think the last entry here was saving the changes I made after the problem happened.

[caenrigen]

Thank you for checking and the updates.

There is definitely at least one nasty bug in there (regarding the lack of a locking mechanism). I will see when I can find the time to look into it, maybe this weekend. Feel free to ping me again next week if needed.

---

sorry I wish they had timestamps

That is actually something that I might be able to fix, TBD

[caenrigen]

@td-anne I published v1.1.6 with a potential fix to prevent concurrent jupytext --sync commands. Give it a try and let me know please if it improves things for you

[caenrigen]

[Note for future readers]

I recommend to additionally upgrade to the latest version of your VSCode(-based) IDE. I was using Cursor v1.1.3 and noticed that a notebook was being saved after <~1s of switching to a Notebook Tab without auto save or auto format being enabled. Very strange. I did not not dig deeper. Updating to Cursor v1.2.2 fixed the problem for me. It is based on VSCode Version: 1.99.3.

[caenrigen]

@td-anne closing the issue for now, open it again if the issue persists for you

[td-anne]

I'm afraid the issue persists, same symptoms. (I don't have permission to reopen the issue.)

[caenrigen]

If you can drop some Jupytext Sync logs again, I added some code to improve understanding what triggered the sync operations

[td-anne]

onDidSaveNotebookDocument,0pxk: [...].ipynb
onDidSaveNotebookDocument,0pxk: Read paired formats for '[...].ipynb': 'ipynb,py:percent'
onDidSaveNotebookDocument,0pxk: Running jupytext sync for [...].ipynb
onDidSaveNotebookDocument,y6i5: [...].ipynb
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Unchanged [...].ipynb
[jupytext] Updating [...].py
onDidSaveNotebookDocument,0pxk: Completed jupytext sync for [...].ipynb
onDidSaveNotebookDocument,y6i5: Read paired formats for '[...].ipynb': 'ipynb,py:percent'
onDidSaveNotebookDocument,y6i5: Running jupytext sync for [...].ipynb
[jupytext] Reading [...].ipynb in format ipynb
[jupytext] Loading [...].py
[jupytext] Updating [...].ipynb
[jupytext] Updating the timestamp of [...].py
onDidSaveNotebookDocument,y6i5: Completed jupytext sync for [...].ipynb

[divenex]

I can confirm this issue with Jupytext Sync v1.1.7. I also receive the "file is newer" error message:

Failed to save 'example.py': The content of the file is newer. Please compare your version with the file contents or overwrite the content of the file with your changes.

Here is the relevant log output:

onDidSaveTextDocument,XXXX: example.py
onDidSaveTextDocument,XXXX: Read paired formats for 'example.py': 'py:percent,ipynb'
onDidSaveTextDocument,XXXX: Running jupytext sync for example.py
[jupytext] Reading example.py in format py
[jupytext] Loading example.ipynb
[jupytext] Updating example.ipynb
[jupytext] Updating example.py
onDidSaveTextDocument,XXXX: Completed jupytext sync for example.py

The key issue seems to be that Jupytext reports "Updating example.py" at the end of the sync process, even when the sync was triggered by saving example.py and the paired notebook (example.ipynb) was unchanged.

This unnecessary write operation on the source file appears to be the root cause of the race condition, leading to the error. This suggests the issue lies within Jupytext itself rather than this extension.