How to do the conversion from TextMate grammar?

[Viyetanami]

I saw your comment on #402 (comment) and I have read the newly added lbstanza.cpp.

I want to know how to convert stanza.tmLanguage.json to lbstanza.cpp. Is there an automated tool?

[SpartanJ]

With the usual tool, as I said ecode now understands TextMate grammars:
ecode --convert-lang-path=/path/to/grammar.tmLanguage.json --convert-lang-output=/output/path/

This step is not needed at all, conversion to C++ is done to integrate the grammar into ecode build (and to load it faster), you don't need to do that. Just put your grammar into the languages folder, it will overwrite any existing grammar if the grammar already exists in the ecode build.

[ghost]

And this becomes a mess now. I don't even know which dialect of syntax highlighting rules is being used in .cpp files. The Lua patterns one is easy to spot. But Lua patterns mixed with RegEx and TextMate grammar (also full of RegEx) are almost the same.

[SpartanJ]

And this becomes a mess now.

How so? This did not remove any feature, I'm iterating over the existing feature, there's no mess, but you can develop much more complex syntaxes with ease given that you can stack-contexts easier than before (I don't think you'll know what I'm talking about because there's no documentation yet).

I don't even know which dialect of syntax highlighting rules is being used in .cpp files.

What do you even mean??? There's a single dialect and nothing changed drastically. There's only the "ecode" grammars. TextMate grammars are being converted on the fly if needed.

The Lua patterns one is easy to spot. But Lua patterns mixed with RegEx and TextMate grammar (also full of RegEx) are almost the same.

You're just talking from your confusion, there's nothing new, Lua Patterns AND RegExs are flagged the same way as before, literally nothing changed. TextMate grammars will just use RegExs.

[ghost]

Another side effect is no one will continue to write in the Lua patterns one anymore. TextMate is superior and more portable.

[SpartanJ]

Another side effect is no one will continue to write in the Lua patterns one anymore.

I'll continue writing with Lua Patterns, they are faster.

TextMate is superior and more portable.

ecode grammars will have all the TextMate capabilities + some own capabilities not present in TextMate (for ex: Lua Patterns + native patterns). So no, TextMate grammars will be used to facilitate users to user their favorite language without having to wait for an ecode official implementation, I'm giving users flexibility and freedom.

[ghost]

Just put your grammar into the languages folder, it will overwrite any existing grammar if the grammar already exists in the ecode build.

If I have both the Lua patterns one (language.json) and the TextMate (language.tmLanguage.json), which will take precedent?

[SpartanJ]

It will behave the same as having the same language definition twice, it will prevail the last loaded, there's no special logic here. language.json or language.tmLanguage.json are the same for ecode, the name doesn't matter actually, TextMate grammars do have a schema definition to identify themselves, so ecode will just load the JSON depending on the schema and format of the JSON file. Oficial format will always be ecode grammars format given that it has some extra features that TextMate does not provide, and also ecode grammars will not be an extension of TextMate, it's just that happens to support them as part of its feature-set but there's a single grammar design / system for ecode.

[Viyetanami]

It will behave the same as having the same language definition twice, it will prevail the last loaded, there's no special logic here.

Hi. But what will be the last loaded?

[SpartanJ]

It's non-deterministic unless you know: the OS, the file system and the POSIX implementation of opendir and readdir. So I cannot answer it. It could be sorted but there's no reason to do it. If you're curious remember that the source code is available.

[Viyetanami]

Is VSCode's language-configuration.json supported?

For example:

https://github.com/covscript/covscript-vscode/blob/master/language-configuration.json

[Viyetanami]

Copy this language-configuration.json into languages directory together with covscript.tmLanguage.json and rename it into covscript-language-configuration.json?

[SpartanJ]

Is VSCode's language-configuration.json supported?

No. At least not for the moment.

Copy this language-configuration.json into languages directory together with covscript.tmLanguage.json and rename it into covscript-language-configuration.json?

Currently it's only supported to load the tmLanguage files (I mean, JSON files with that particular format). Name can be anything, it only needs to have a .json extension.

[Viyetanami]

The TextMate grammar file name is not covscript.tmLanguage.json, do I need to rename it?

https://github.com/covscript/covscript-vscode/blob/master/syntaxes/covscript.cson.json

[SpartanJ]

cson is not supported but that file is actually a converted cson to json file, it seems to be valid, although highlighting looks like crap with that file, it seems that they use non-standard scope names which they are all translated to just "normal" color, if it's only that it's trivial to adapt to ecode, you can basically load that grammar using that json and run:
ecode --export-lang=covscript --export-lang-path=/export/path/covscript.json

And then modify the exported grammar (which will be in ecode format) and replace the "normal" types to some colored option. And then replace the original textmate grammar with the ecode new modified grammar.

[Viyetanami]

Quoted your comment on #402 (comment):

Added. It's a direct conversion from the TextMate grammar you shared. ecode has now experimental support for TextMate grammars. tmLanguage.json files can be moved to the languages folder and they will be loaded directly. There are some limitations, grammars that depend on other grammars will not work, some complex cases might not work, but it's a good starting point, many grammars will work directly with no intervention. But, there's a minor catch, some definitions do not expose the file extensions that open those files, in those cases it's needed to modify the language definition to specify this in the root JSON object of the grammar: "fileTypes": ["ext", "any_other_ext"]. ecode grammars documentation will be updated soon.

It seems the file extensions are covered in package.json.

For example:

https://github.com/objeck/objeck-lang/blob/master/docs/syntax/vscode/package.json

[SpartanJ]

Yes, but I'm not planning to support TextMate bundles, at least for the moment, only the grammars and mostly to help with the grammars adaptation / port, I still want to have our own grammars and not limit the format to TextMate, having multiple files to load a single language is inefficient, I don't like it.
The TextMate grammars spec allows to specify the file extensions as I presented, so at least that can be used.

[Viyetanami]

objeck.tmLanguage.json gives no syntax highlighting at all. What's going on?

https://github.com/objeck/objeck-lang/blob/master/docs/syntax/vscode/syntaxes/objeck.tmLanguage.json

[Viyetanami]

Fixed by adding "fileTypes": ["obs"], after "name": "Objeck", in objeck.tmLanguage.json.

[SpartanJ]

Just FYI : Objeck is already supported by ecode, I just opened a few obs files and they open fine, not sure how did not highlight in a first place.

[Viyetanami]

I know that Objeck is already supported. I just pick randomly languages with VSCode syntax highlighting in the syntax highlight tag of the Issues page to test. Frankly speaking, I think objeck.tmLanguage.json looks worse than the built-in support for Objeck. I don't know how it looks like natively on VSCode, though. I don't have VSCode to test.

[SpartanJ]

That's just because it's worst, it's quite incomplete by just looking at the implementation.