Extra tokens from CSS with SWC

[romainmenke]

I would like to compare the CSS Tokenizer in SWC to other tools.

https://github.com/romainmenke/css-tokenizer-tests

The goal is to verify that multiple tokenizers correctly implement the specification.

For example :

• which tools preserve comments
• which tools support escaped code points
• ...

Ultimately I hope that this will improve the overal quality of CSS Tokenizers in use by the wider community.

I think it is likely that the SWC tokenizer is the most accurate and usable can serve as a good reference to others.

---

Is there a way to do this :

1. take a bit of CSS
2. tokenize the CSS with the SWC tokenizer
3. expose the tokens to JavaScript (writing structured data to a file or stdout is fine)

I don't know anything about rust so it's not easy for me to determine if this is feasible and how to do it.

A small rust program that reads CSS from stdin and writes token data to stdout would be ideal.