Get an array of all tokens

[avigoldman]

Is there a way to get an array of all tokens? Currently I've got the following code:

const tokens = new Tokenizer(str).readTopLevelTokens();

This gives the top-level Tag, Output, and HTML tokens. I'm looking for a why to get a similar array but containing all of the tokens parsed all the way down.

So ideally the end result would be an array of all tokens (Number,Literal,Tag,Output,HTML,Filter,Hash,PropertyAccess,Word,Range,Quoted,Operator)

[harttle]

duplicate with #528, I'd like to move this to discussion.

The key point is parsing is recursive, and defined by each tag. That means the parsed structure is only known by the Tag itself. Maybe in v10 we can try out a unified parsing and finally define the AST of Liquid language.

[avigoldman]

If we were to recursively flatten each node, would that have the same effect?