Skip to content

Use YAML version traits in the parser #229

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 64 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
1cf7c1e
Split mark and span from `src/tokens.jl`
Paalon Jun 14, 2024
60a1ece
Add iteration and indexing to `Span`.
Paalon Jun 14, 2024
f93dbb2
Add methods to get start mark and end mark from tokens.
Paalon Jun 14, 2024
70698b3
Replace `token.span.start_mark` and `token.span.end_mark` to
Paalon Jun 14, 2024
b4cd9f1
Fix existing bugs about getting marks from tokens.
Paalon Jun 14, 2024
6efb3d5
Replace remained expressions.
Paalon Jun 14, 2024
1c9a97d
Refactoring of `scan_line_break`.
Paalon Jun 15, 2024
e9c5827
Rewrite `forwardchars!(::TokenStream, ::Integer)`.
Paalon Jun 17, 2024
a9d3d1a
Change to use performant and explicit function for whitespace checking.
Paalon Jun 17, 2024
a56bdce
Use `yaml_1_1_is_whitespace` where not used.
Paalon Jun 17, 2024
dba3e68
Merge branch 'master' into scan_line_break
Paalon Jun 17, 2024
69645b5
Merge branch 'master' into mark-span
Paalon Jun 17, 2024
76a2aa5
Change `forward!(::BufferedInput, ::Integer)` to explicitly return `n…
Paalon Jun 17, 2024
aa04dae
Change `_fill` and `__fill` to better implementation and rename to
Paalon Jun 17, 2024
c4ef02d
Bug fix of `prefix(::BufferedInput, ::Integer)`.
Paalon Jun 17, 2024
1595c5b
Merge branch 'master' into scan_line_break
Paalon Jun 18, 2024
5d6632c
Make it explicit that scalar addition then vector addition.
Paalon Jun 18, 2024
ab9b7dd
Merge branch 'bi-buffer' into bi-3
Paalon Jun 18, 2024
f710098
Merge branch 'master' into bi-3
Paalon Jun 18, 2024
76f857b
Merge branch 'master' into scanner-whitespace
Paalon Jun 18, 2024
787111e
Merge branch 'master' into scanner-whitespace-more
Paalon Jun 18, 2024
76bcbcc
Merge branch 'master' into bi-nothing
Paalon Jun 18, 2024
a6c6d8f
Merge branch 'master' into mark-span
Paalon Jun 18, 2024
6d7dd03
Remove `equivalent` and change to `isequal` and remove chomping.
Paalon Jun 18, 2024
9f4fe51
Merge branch 'master' into mark-span
Paalon Jun 18, 2024
dd87992
Merge branch 'master' into bi-3
Paalon Jun 19, 2024
7cf0234
Merge branch 'master' into bi-nothing
Paalon Jun 19, 2024
1db5b78
Merge branch 'master' into bi-nothing
Paalon Jun 20, 2024
0d705dd
Merge branch 'master' into mark-span
Paalon Jun 20, 2024
acfb2b3
Remove methods for `Span`.
Paalon Jun 20, 2024
73a52ff
Add objects to represent YAML versions.
Paalon Jun 20, 2024
3dbeadf
Use YAML version traits for `b-char`.
Paalon Jun 21, 2024
689e27a
Merge branch 'b-char' into scan_line_break
Paalon Jun 21, 2024
6738103
Use better implementation because the document iterator bug has been
Paalon Jun 21, 2024
8542039
Use YAML version traits for `scan_line_break`.
Paalon Jun 21, 2024
3983d9c
Merge branch 'yamlversion' into scanner-whitespace
Paalon Jun 21, 2024
d3358b9
Merge branch 'b-char' into scanner-whitespace
Paalon Jun 21, 2024
1f329f2
Use YAML version traits for `is_whitespace` and move its definition t…
Paalon Jun 21, 2024
331c5a2
Merge branch 'scanner-whitespace' into scanner-whitespace-more
Paalon Jun 21, 2024
78c04b3
Change remained `yaml_1_1_is_whitespace` to use YAML version traits.
Paalon Jun 21, 2024
af6dc18
Merge branch 'b-char' into y11-forwardchars
Paalon Jun 21, 2024
be9d213
Use YAML version traits for `forwardchars!`.
Paalon Jun 21, 2024
427158f
Refactoring and sort out functions.
Paalon Jun 21, 2024
0d82a13
Merge branch 'master' into bi-3
Paalon Jun 21, 2024
a1a293f
Merge branch 'master' into test-isequal
Paalon Jun 21, 2024
d9b45af
Rename `get_mark` to `Mark`.
Paalon Jun 21, 2024
fe68efb
Merge branch 'master' into scan_line_break
Paalon Jun 21, 2024
5378896
Merge branch 'master' into scanner-whitespace
Paalon Jun 21, 2024
a13f9ed
Merge branch 'scanner-whitespace' into scanner-whitespace-more
Paalon Jun 21, 2024
36ed962
Merge branch 'master' into y11-forwardchars
Paalon Jun 21, 2024
074b04c
Merge branch 'master' into mark-span
Paalon Jun 21, 2024
01db7c0
Merge branch 'master' into test-isequal
Paalon Jun 21, 2024
fa885a0
Merge branch 'master' into rm-get-mark
Paalon Jun 22, 2024
28004f7
Merge branch 'master' into bi-nothing
Paalon Jun 22, 2024
22251c9
Merge branch 'master' into bi-3
Paalon Jun 22, 2024
9a67da0
Merge branch 'scan_line_break' into main-scan
Paalon Jun 22, 2024
beecaec
Merge branch 'y11-forwardchars' into main-scan
Paalon Jun 22, 2024
a753635
Merge branch 'rm-get-mark' into dev-scan
Paalon Jun 22, 2024
4204b0e
Merge branch 'test-isequal' into dev-scan
Paalon Jun 22, 2024
1db2624
Use YAML version traits for most functions in `src/scanner.jl`.
Paalon Jun 22, 2024
b97e9e3
Merge branch 'mark-span' into dev-parser
Paalon Jun 22, 2024
f6067a0
Merge branch 'bi-nothing' into dev-parser
Paalon Jun 22, 2024
33f9ea5
Merge branch 'bi-3' into dev-parser
Paalon Jun 22, 2024
bf84c85
Use version traits for the parser.
Paalon Jun 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/YAML.jl
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ using StringEncodings
include("versions.jl")
include("queue.jl")
include("buffered_input.jl")
include("mark.jl")
include("span.jl")
include("tokens.jl")
include("scanner.jl")
include("events.jl")
Expand Down
25 changes: 10 additions & 15 deletions src/buffered_input.jl
Original file line number Diff line number Diff line change
Expand Up @@ -15,41 +15,35 @@ mutable struct BufferedInput
end
end


# Read and buffer n more characters
function __fill(bi::BufferedInput, bi_input::IO, n::Integer)
for _ in 1:n
c = eof(bi_input) ? '\0' : read(bi_input, Char)
i = bi.offset + bi.avail + 1
# Read and buffer `n` more characters
function buffer!(bi::BufferedInput, n::Integer)::Nothing
for i in (bi.offset + bi.avail) .+ (1:n)
c = eof(bi.input) ? '\0' : read(bi.input, Char)
if i ≤ length(bi.buffer)
bi.buffer[i] = c
else
push!(bi.buffer, c)
end
bi.avail += 1
end
bi.avail += n
nothing
end

_fill(bi::BufferedInput, n::Integer) = __fill(bi, bi.input, n)

# Peek the character in the i-th position relative to the current position.
# (0-based)
function peek(bi::BufferedInput, i::Integer=0)
i1 = i + 1
if bi.avail < i1
_fill(bi, i1 - bi.avail)
buffer!(bi, i1 - bi.avail)
end
bi.buffer[bi.offset + i1]
end


# Return the string formed from the first n characters from the current position
# of the stream.
function prefix(bi::BufferedInput, n::Integer=1)
n1 = n + 1
if bi.avail < n1
_fill(bi, n1 - bi.avail)
end
function prefix(bi::BufferedInput, n::Integer=1)::String
bi.avail < n && buffer!(bi, n - bi.avail)
String(bi.buffer[bi.offset .+ (1:n)])
end

Expand All @@ -70,6 +64,7 @@ function forward!(bi::BufferedInput, n::Integer=1)
n -= 1
end
end
nothing
end

# Ugly hack to allow peeking of `StringDecoder`s
Expand Down
10 changes: 10 additions & 0 deletions src/mark.jl
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Position within the document being parsed
struct Mark
index::UInt64
line::UInt64
column::UInt64
end

function show(io::IO, mark::Mark)
@printf(io, "line %d, column %d", mark.line, mark.column)
end
Loading
Loading