fix: do not clamp character recvd from client

[heywhy]

This fixes #110 and #83.

It was observed that the client sends the correct value for character, but Expert clamps the value to the size of the line the edit falls on, which is usually incorrect behavior for clients like Neovim that batch multiple changes into a single request to the language server. Using the below as an example:

defmodule Proxy do
  String.
end

Due to batching, Neovim sends a textDocument/didChange notification with multiple content changes:

[
  {
    "text": "b",
    "rangeLength": 0,
    "range": {
      "start": {"line": 1, "character": 9},
      "end": {"line": 1, "character": 9}
    }
  },
  {
    "text": "y",
    "rangeLength": 0,
    "range": {
      "start": {"line": 1, "character": 10},
      "end": {"line": 1, "character": 10}
    }
  },
]

So, the second edit has its character truncated to 10 instead of the 11 that the client sent, and we end up with the below as a result.

defmodule Proxy do
  String.yb
end

[mhanberg]

I think the problem might be that GenLSP (the library powering the LSP protocol) is processing the text document did change notifications out of order, as they are handled asynchronously.

I think they need to be special cased and handled in order.

I am out of town and haven't tested that yet, but I wanted to let you know in case you want to pause work on this until I validate my hypothesis.

[heywhy]

I doubt GenLSP is the problem, as I still get the below in a recent log.

%XPGenLSP.Structures.DidChangeTextDocumentParams{
  content_changes: [
    %{range: XpRange[<<7, 36>>...<<7, 37>>], text: "", range_length: 1},
    %{range: XpRange[<<7, 35>>...<<7, 36>>], text: "", range_length: 1},
    %{range: XpRange[<<7, 34>>...<<7, 35>>], text: "", range_length: 1},
    %{range: XpRange[<<7, 33>>...<<7, 34>>], text: "", range_length: 1},
    %{range: XpRange[<<7, 32>>...<<7, 33>>], text: "", range_length: 1},
    %{range: XpRange[<<7, 31>>...<<7, 32>>], text: "", range_length: 1},
    %{range: XpRange[<<7, 31>>...<<7, 31>>], text: "_", range_length: 0},
    %{range: XpRange[<<7, 32>>...<<7, 32>>], text: "_", range_length: 0},
    %{range: XpRange[<<7, 33>>...<<7, 33>>], text: "M", range_length: 0}, 
    %{range: XpRange[<<7, 34>>...<<7, 34>>], text: "O", range_length: 0},
    %{range: XpRange[<<7, 35>>...<<7, 35>>], text: "D", range_length: 0},
    %{range: XpRange[<<7, 36>>...<<7, 36>>], text: "U", range_length: 0},
    %{range: XpRange[<<7, 37>>...<<7, 37>>], text: "L", range_length: 0},
    %{range: XpRange[<<7, 37>>...<<7, 37>>], text: "E", range_length: 0},
    %{range: XpRange[<<7, 37>>...<<7, 37>>], text: "_", range_length: 0},
    %{range: XpRange[<<7, 37>>...<<7, 37>>], text: "_", range_length: 0}
  ], 
  text_document: %XPGenLSP.Structures.VersionedTextDocumentIdentifier{uri: "file:///Users/heywhy/Dev/me/proxy/lib/proxy.ex", version: 236}
}

This is not a GenLSP problem as I've pointed out in #110 (comment) . Also, I've added a screen recording to the PR description.

[mrluc]

@heywhy oh hey, this is great -- your updated description, re: 'clamping', and dc8bcce, feels very promising.

I understand your hypothesis to be "it's clamping to the line as we knew it when the message was received, which is only guaranteed to work for the first edit in batched-together edits"

So if that's the issue, @mhanberg WDYT about value/lack of value of 'clamping'? This solution reads to me like "clamping is a bad idea, let's just trust that the client gave us a sensible position". But of course, if the line was updated after each edit was applied in that batch scenario, then the existing 'clamped' logic would work too no?

[heywhy]

But of course, if the line was updated after each edit was applied in that batch scenario, then the existing 'clamped' logic would work too no?

The existing clamping logic won’t work because there can be multiple edits that increase and the same time decrease the size of the line. And considering the fact that the values have already been truncated (reduced to the current size of the line) even before applying any of the edits, then the in-memory representation of the document gets thrown into an incorrect state.

For instance, the current size of line is 5 (abcde), there are edits that extend the line beyond 5 characters (fghij) and also delete 6 characters afterwards (jihgh). Due to the existing clamping logic, no deletion happens because the start and end character have been reduced to 5 and it results to 0 characters to be deleted.

[mhanberg]

This change makes a lot more sense to me now, I am still confused why it didn't seem to be a problem with lexical, but maybe in a recent name of them update they changed how this works.

Could you try and add a test?

[heywhy]

Done.