Binary parsing

[vaguue]

Hey there! Some time ago I made a parser for C structs using Chevrotain, as a utility for my project over-the-wire, to parse network packets and handle some other low-level tasks (but let’s focus on parsing and building network packets in this discussion). It worked well for a while, but at some point I realized that parsing plain C structs wasn’t enough — for example, parsing a DNS packet involves conditional fields, and TLV values are not straightforward to parse, especially considering there are multiple layers in the buffer, etc.
So I started looking again for JavaScript binary parsing libraries, but none of them seem to fit my current requirements. Here’s what I’m looking for right now:

1. Something resembling Zod in its interface — I think many of us are used to Zod, so having a Zod-like experience for binary formats would be great.
2. Cosmetic features like declaring a field as an IPv4 or MAC address.
3. Lazy parsing: new DNS(buf) should not eagerly read everything; it should expose methods/getters that parse fields on demand (as my original struct-compile does).
4. Support for nested structures and bitfields.
5. A flavour closer to monadic parser combinators

I tried to write some code with this imaginary library, and it looks something like this:

import { struct, u8, u16, u32, bytes, array, ref, choice, lazy } from 'binarrow';

const Tlv = struct({
  type:   u8,
  length: u16,
  value:  bytes(ref('length')),
});

const IPv4 = struct({
  versionIhl: u8.bitfield(4),
  dscpEcn:    u8.bitfield(4),

  totalLen:       u16.be,
  identification: u16.be,
  flagsFragOff:   u16.be,

  ttl:        u8,
  protocol:   u8,
  hdrChecksum:u16.be,

  srcIp: u32.be.as('ipv4'),
  dstIp: u32.be.as('ipv4'),

  options: bytes(
    ref('versionIhl')
      .bitAnd(0x0f)
      .mult(4)
      .minus(20)
  ).refine(Tlv),
});

const buf = Buffer.from(...);
const hdr = new IPv4(buf);

console.log(hdr.ttl);        // lazy read
console.log(hdr.options);    // parsed TLVs
console.log(hdr.toObject()); // materialise everything

I’m also thinking about conditional and choice combinators, etc. This is a very rough draft of what I’m aiming for. In Rust there’s the nom library, which I find inspiring, and Haskell’s parsec as well. But the API design is still a work in progress.

Regarding implementation, my current library has poor performance, so I’m considering using new Function to compile the parsers while keeping a declarative, combinator-style syntax.

I know this is kind of off-topic for Chevrotain, but since I think the author of Chevrotain understands parsers in JavaScript better than anyone else, I don’t know a better place to ask for advice.

[vaguue]

And my questions are:

1. What do you think about the API design in general?
2. What are the places to look at?
3. Is generating a string with a code for parsing during initialization and calling new Function/eval is a good idea?