Feature request: more generalized PrefixUpTo?

[junebash]

In one of my projects, I have a line in a parser that looks like this:

OneOf {
  PrefixUpTo("\n\n".utf8)
  PrefixUpTo("\r\n\r\n".utf8)
  Rest()
}

Really what I need is to parse everything up to the next two lines, or the rest of the string otherwise, with something like this:

OneOf {
  PrefixUpTo {
    Whitespace(2, .vertical)
  }
  Rest()
}

This isn't possible to express currently. Seems like it should be possible?

I'll see if I can cook up a PR, but I figured I'd mention it here first.

[stephencelis]

@junebash The main reason we've resisted adding this directly to the repo is a theoretical worry about performance...if the parser passed to PrefixUpTo is expensive, and runs on every character/byte till it matches, this could lead to very slow parsers. With a static sequence we can make some better performance guarantees.

It may be the case that our worry is unjustified, though! We'd love to see benchmarks comparing the approaches and trying to see if it's possible to encounter an issue.

One other thought is with Regex looming, this kind of functionality could be filled by some Regex helpers:

OneOf {
  PrefixUpTo(/\v\v/)
  Rest()
}

I think this (and other things) will be fun to explore soon!

Also, I'm going to convert this to a discussion, since we try to keep issues focussed on bugs.

[junebash]

@stephencelis Just as you posted this I was experimenting with trying to implement it and realized the very issue you mentioned. 😅 I hadn't thought of Regex being able to help here; that will definitely be interesting! Regardless, thanks for chiming in.

[stephencelis]

@junebash While it definitely feels like it might be a problem to repeatedly call .parse while consuming an element at a time, in practice most parsers should fail very quickly, so I'd definitely say it's worth trying out and benchmarking. If it turns out to not be a problem we should really consider introducing it directly to the library!

[mackoj]

@junebash I have open a PR with a more generalised PrefixUpTo #229 it fast as fast as PrefixUpTo in most cases 😁