Trigger resolveAll when eof is reached

[JulesGuesnon]

Hello!

First of all, thanks a lot for this awesome library!

I am currently trying to implement an extension to parse custom html tags and then turn them into custom AST, e.g: <myTag> and </myTag> (I don't want to handle auto-closing tag for now).
I could easily implement the parsing for both tokens, but I am having a problem with the resolution. In the resolveAll, I implemented a detection for tags that are alone, and I turn them into data if that's the case.

My understanding was that resolveAll was being triggered when all the events were produced, but it looks like it's not the case, and I am detecting token that are not closed while it's actually the case, it's just that the closing token hasn't been tokenized yet.

Thus the title of the discussion, I was wondering if there is a way to trigger the resolveAll only when the eof is reached?
The thing is, I may be approaching the problem the wrong way, so maybe there is another way of doing what I want to do?

[wooorm]

Hi! A minimal reproduction would help. I believe resolveAll should indeed be called on eof. If it isn’t, there is likely a reason for it. First thing I think of: the construct never parses anything to ok, and thats why its corresponding resolveAll isn’t called

[JulesGuesnon]

Thanks a lot for the quick answer!

Here is a minimal repro (sorry the resolveAll is pretty big):

import { parse, postprocess, preprocess } from "micromark";
import { splice } from "micromark-util-chunked";
import { codes } from "micromark-util-symbol";
import type {
  ConstructRecord,
  Extension,
  Resolver,
} from "micromark-util-types";

const resolveAll: Resolver = (events, context) => {
  let closeI = -1;

  while (++closeI < events.length) {
    const currentClose = events[closeI];

    if (currentClose[0] !== "enter" || currentClose[1].type !== `closeTagTemp`)
      continue;

    let openI = closeI;

    while (openI--) {
      const currentOpen = events[openI];

      if (currentOpen[0] !== "exit" || currentOpen[1].type !== `openTagTemp`)
        continue;

      currentOpen[1].type = `openTag`;
      currentClose[1].type = `closeTag`;
    }
  }

  closeI = -1;

  while (++closeI < events.length) {
    const current = events[closeI];

    if (current[1].type === `closeTagTemp`) {
      current[1].type = "data";
    }

    if (current[0] !== "enter" || current[1].type !== `openTagTemp`) continue;

    let exitI = closeI - 1;

    while (++exitI < events.length) {
      const currentExit = events[exitI];

      if (currentExit[0] !== "exit" || currentExit[1].type !== `openTagTemp`) {
        continue;
      }

      const newToken: Token = {
        type: "data",
        start: currentExit[1].start,
        end: currentExit[1].end,
      };

      splice(events, closeI, exitI - closeI + 1, [
        ["enter", newToken, context],
        ["exit", newToken, context],
      ]);

      break;
    }
  }

  return events;
};

const tokenizers: ConstructRecord = {
  [codes.ampersand]: {
    name: `open`,
    tokenize: (effects, ok) => {
      return (code) => {
        effects.enter("openTagTemp");
        effects.exit("openTagTemp");
        effects.consume(code);

        return ok(code);
      };
    },
    resolveAll,
  },
  [codes.exclamationMark]: {
    name: `close`,
    tokenize: (effects, ok) => {
      return (code) => {
        effects.enter("closeTagTemp");
        effects.exit("closeTagTemp");
        effects.consume(code);

        return ok(code);
      };
    },
    // resolveAll,
  },
};

const ext: Extension = {
  text: tokenizers,
  flow: tokenizers,
};

console.log(
  postprocess(
    parse({ extensions: [ext] })
      .document()
      .write(preprocess()(`&Hello!`, undefined, true)),
  ).map(([a, b]) => [a, b.type]),
);

If I don't add the resolveAll, this is the output:

[
  [ "enter", "openTagTemp" ], [ "exit", "openTagTemp" ], [ "enter", "lineEnding" ], [
    "exit", "lineEnding"
  ], [ "enter", "content" ], [ "enter", "paragraph" ], [ "enter", "data" ], [ "exit", "data" ], [ "enter", "closeTagTemp" ], [ "exit", "closeTagTemp" ]
, [ "exit", "paragraph" ],
  [ "exit", "content" ]
]

From the moment I add the resolveAll, I get the following:

[
  [ "enter", "data" ], [ "exit", "data" ], [ "enter", "lineEnding" ], [ "exit", "lineEnding" ], [ "enter", "content" ], [ "enter", "paragraph" ], [ "en
ter", "data" ], [ "exit",
    "data"
  ], [ "enter", "data" ], [ "exit", "data" ], [ "exit", "paragraph" ], [ "exit", "content" ]
]

Now, if you add a console.log in the in the resolveAll, it looks like my token does not exist when the one of the open is called:

[
  [ "enter", "openTagTemp" ], [ "exit", "openTagTemp" ], [ "enter", "lineEnding" ], [
    "exit", "lineEnding"
  ], [ "enter", "content" ], [ "enter", "paragraph" ], [ "enter", "chunkText" ], [ "exit",
    "chunkText"
  ], [ "exit", "paragraph" ], [ "exit", "content" ]
]

Same when it's called in the closed one:

[
  [ "enter", "data" ], [ "exit", "data" ], [ "enter", "closeTagTemp" ], [ "exit", "closeTagTemp" ]
]

Sorry for the super long message, it looks like I am likely misunderstanding the resolveAll and when it's being called