RFC: Organize the public API of swc_ecma_lexer and swc_ecma_parser

[CPunisher]

Background

Since #10399, there has been two sets of lexers and parsers in swc projects:

• The lexer and parser in swc_ecma_lexer for compatibility. There are also all common lexer and parser logic and traits.
• The lexer and parser in swc_ecma_parser for performance. This is what we are still actively improving their performance.

Sadly, it's hard to immediately remove the old-fashion lexer and parser:

1. Since we have made the old lexer public, downstream crates like rspack depends on it.
2. Although I think it's not appropriate to expose any lexer, due to the problem 1, we have to provide alternatives for the users, which means we also have to make the new lexer public.
3. Based on the problem 1 and 2, we should make sure that swc_ecma_parser is the exact equivalent of swc_ecma_lexer. It could be hard to prove because swc_ecma_parser doesn't have strict token validation as we discussed in refactor(es/parser): Split parser into also-lex/parse-only #10399

Note that there's only fast-strip crate in swc projects that depends on the old-fashion lexer and parser, but it has been proved possible and worth to migrate, see #10677.

Motivation

1. Make fast-strip benefit from our new performant lexer and parser.
2. Make swc project less chaotic
3. (Not sure) encourage those depending on old-fashion lexer and parser to migrate to the new faster lexer and parser.

Design Details

Even if we are only considering the performance of projects in swc itself, due to the usage of the lexer in fast-strip, we have to make lexer api in swc_ecma_parser public.

Based on the idea above, I have come up with some options:

1. Make lexer api in swc_ecma_parser public for fast-strip. But if we don't want to make it public to all downstream applications we can mark it with an internal feature and add documents as said in fix: change visibility of lexer::token exports from crate to public #10537 (comment)
2. If we have confidence that swc_ecma_parser is an exact equivalent of swc_ecma_lexer, we may make it public to all and deprecate the old-fasion lexer and parser in the future.
   • Risk: this could be hard and bring more chaos to the ecosystem during the migration stage. And it will be worse if there's any corner case that the new lexer and parser cannot handle.

[kdy1]

I think we should encourage users to migrate to fast parser anyway, because the main difference is correctness of TokenKind, and it's somewhat inferrable from the user side.

[CPunisher]

I think we can currently add an unstable feature for the new lexer, just like what other crates do (tokio_unstable, parking_lot_nightly), which means there could always be some breaking changes with that (although I suppose there won't be). For those caring about the performance, they can choose to migrate their lexers and tokens.

[kdy1]

I like the idea. The only thing is that we should clearly document that it is not semver compilant

[CPunisher]

Closed as #10744