Parsing a Python file with no trailing newline

[nchammas]

I've been struggling with a corner case in my DSL built using Lark. It has significant indentation like Python.

The corner case is when the program text happens not to end with a newline. I kept trying to massage my rules and terminals to handle this case cleanly, until I took a look at Lark's implementation of the Python grammar for inspiration and came across this:

lark/examples/advanced/python_parser.py

Line 59 in 87bb8ef

chosen_parser.parse(r + '\n')

Is this basically the "correct" way to cover this case in Lark?

I noticed that in CPython they have something called ENDMARKER, which I suppose solves this in a different way, though I don't understand the details:

https://github.com/python/cpython/blob/c81fa2b9cd1ed4d4ff5c13984d6582c27e5f2633/Doc/reference/toplevel_components.rst?plain=1#L72

[erezsh]

I consider it correct, as it's the simplest, and is equivalent to EOF without requiring a special marker.

There is an open PR I made long ago that tried to implement support for EOF, but it had subtle issues, and I decided not to merge it in.