Ignore Scopes in word count [including proof of concept]

[mbroedl]

Resulting from a discussion with @X-Raym in the recently accepted PR #94 I had a bit of a thought on why it should not be possible to ignore scopes rather than re-inventing regexes that are implemented as grammars already.

Taking tokenised lines is theoretically possible. Together with a scope selector it is possible to filter (positively or negatively) all text elements that match certain scopes.
This would further allow to ignore punctuation (much requested although now dealt with in a new word count regex), different grammars, etc (see e.g. #55 and #65 ).

See the following minimal example to filter out all scopes that are comments, quotes, or punctuation in their respective grammar. It should be copyable to the console of atom 1:1.

let scopes = ['comment.*', 'quote.*', 'punctuation.*'];
let editor = atom.workspace.getActiveTextEditor();
let scopeselector = require('first-mate').ScopeSelector;

let buffer = [];
editor.displayBuffer.tokenizedBuffer.tokenizedLines.forEach(line => line.tokens.forEach(token => buffer.push(token)));

scopes.forEach(scope => {
    let selector = new scopeselector(scope);
    buffer = buffer.filter(token => !selector.matches(token.scopes));
});

let text = buffer.map(token => token.value).join('');
console.log(text.match(/\S+/g).length);

At the moment it reduces all lines into a single buffer to make looping easier. this destroys line breaks, and may thus not desirable, but I thought as a proof of concept this would be sufficient for now. This does change the final word count a little bit though. — It should be no difficulty to instead loop through the tokens and filter those that do not match the ignored selectors.

Maybe instead of the many current settings, one could have one text box where all to-be-ignored scopes are listed, and then a little pop-up menu on right-click on the word count (similar to the one for the minimap) would allow to activate/deactivate certain scopes in filtering.
This would reduce bulk in the settings (see discussions elsewhere in this package).

Disclaimer:
(A) The tokenised lines are not documented and thus subject to change without further notice, although the current selector seems to have been stable for at least two years.
(B) I do not know how much the speed of calculation would suffer from this way of filtering (compared to regex, and compared to not filtering).

Looking forward to any discussions!

[mbroedl]

This code is a bit nicer. Now keeping line breaks.

 // ignoring comments, quotes, and fenced code, not ignoring punctuation
let scopes = ['comment.*', 'quote.*', '!punctuation.*', 'fenced.code.*'];
let editor = atom.workspace.getActiveTextEditor();
let scopeselector = require('first-mate').ScopeSelector;

let selector = new scopeselector(scopes.filter(s => !s.startsWith('!')).join(', '));

let text = editor.displayBuffer.tokenizedBuffer.tokenizedLines.map(
    line => line.tokens.filter(
        token => !selector.matches(token.scopes)
    ).map(
        token => token.value
    ).join('')
).join('\n');

console.log(text.match(/\S+/g).length);

As the scope selector functions like css selectors (connection with commas), there is two ways to ignore something (or deactivate temporarily, see above): either filter it out from the list before (here done via an exclamation mark), or add a random word such as not, so that 'not punctuation.*' can never be matched and is thus ignored. The first option is cleaner, the second one easier.

[X-Raym]

@mbroedl Also for testing my particular case (usage of this plugin with Fountain, I write a bit of code which exclude things which should be not be considered in time evaluation of .fountain files to see how it could be implemented. It can surely be more clean with more advanced regex, but it gives an idea:

stripText: (text, editor) ->

...


    arr = text.split('\n')
    out = []
    for i of arr
      if arr[i].match(/[a-z]/gmu) && !arr[i].match(/^[=|#]/gmu)
        out.push arr[i]
    out = out.join('\n')
    text = out.replace(/(\/\*[^*]*\*\/)|(\/\/[^*]*)/g, '')

    text

So here my conclusion about that:

Having to maintain a different fork of the plugin just to add minimal code snippet is too bad. Filters right in settings can be good, but having the possibility to extend the filtering with custom code snippet in the init.coffee file will be better. It will allows to have complex functions, and to limit them to certain grammar.

[X-Raym]

I think this concept of hook is called Service Hub.

This is how I extended some Pigments function.

Here is the doc:

https://github.com/atom/service-hub

Or maybe just Services

https://flight-manual.atom.io/behind-atom/sections/interacting-with-other-packages-via-services/

(I don't know if there is a diff)

[mbroedl]

@X-Raym Although I prefer out-of-the-box packages, I see the point of your suggestions.
But could you explain how you would want to use services here?

[X-Raym]

@mbroedl The point would be to be able to let the user write its own text filtering functions in his init.cofee file. I know some package allows such features, like pigments to add custom named colors. Though, I don't know how to implement that.

But regarding to your proposition, it is able to get the syntax of the line, it is not just text filtering right ?

In this case, we could imagine a filter at this level, and a filter at the text variable level (filtering there doesn't allow to get original syntax).

[mbroedl]

I see, yeah. I haven't implemented something like that yet, but it shouldn't be too bad?

Here's the logics in pigments, I suppose:
https://github.com/abe33/atom-pigments/blob/master/package.json#L20
https://github.com/abe33/atom-pigments/blob/master/lib/pigments.coffee#L121
https://github.com/abe33/atom-pigments/blob/master/lib/pigments-api.coffee

Yes, my proposition gets the tokenised syntax (i.e. what your grammar defines, such as fountain, markdown, asciidoc, or whatever) and then filters everything required. So everything you see when syntax highlighting (and whatever is defined but not highlighted).
So comment.* would filter anything that the currently activated grammar tags as a comment, for example comment.md or comment.fountain or comment.py (don't know if the latter exist?). And then you can extend it or specify it more, for example: comment.fountain would only exclude fountain comments but not markdown comments from the wordcount. And speaker.annotation.fountain (I made this one up) could ignore every speaker declaration...

So I suppose if your fountain grammar includes all the regex already and tokenises it properly, there might not even be a need to have a service?

Although there might be cases where you would want to ignore stop words in the word count, or prepositions, but not nouns... ok, that's getting well advanced.

[X-Raym]

@mbroedl I have to admit having tokenized syntax would make filtering far more easy, as I would have have to reimplement regexes at the filter level. So Yes, this can be a very good solution too ! I would only have to put a list of desired syntax in a text field in the settings right ? That would be very quick to set up :) Maybe one text field for adding syntax, and one to exclude some (depending on how many syntax you need, one would be quicker to set up).

ok, that's getting well advanced.

It's getting interested :D Having tokenzied syntax filter would satisfy my immediate needs, a text filter is really for advanced usage.

[mbroedl]

I envisioned something like that on rightclick, although this'd be a lot of effort to implement.

They'd probably be stored as which should be changeable in the settings, as well:
comments.*, !*.todo, quote.*, punctuation.*, fenced.code.*, !header.*

What do you mean with 'a text field for adding syntax'? Like a regex to further exclude something that is not defined in the grammar?

[X-Raym]

@mbroedl oh no I imagine a simple text input fields in word count settings with a csv list of syntax to keep or exclude. But your solution is from far the most user friendly and flexible, I like it !

[mbroedl]

Yeah, I suppose your suggestion would be a first implementation.
And only then comes the UI to it, when it's needed and someone had time to implement it. But that probably can wait way longer.

I wouldn't actually keep two fields, because of the excessive amount of grammars. There'd be too much that's in neither list.

So I would suggest only to have a field to exclude grammars?
At least in my case that would work well, I don't know about you. In markdown there would be no grammar that would be ideally included. As in: everything is text.md but normal text does not have any extra specifications apart from that, and then everything is included, just like it would normally. (Even within fenced code blocks, the text.md grammar is still present!)

What do you think? Do you have any use cases for when it would be reasonable to do an inclusion list instead? (Apart from the activation-grammar list that already exists?)

[X-Raym]

@mbroedl in fountain there is only few fields which are valuable for wordcount : dialog text, description, and transition. The rest have to be exluded. But fortunatly there isnt too much syntax element so it will be possible to do this with an exclusion list. Longer but possible.

But I think Inclusion list is thesafest choice. It gives less surprise. "Just put there what you want", no surprise with syntax which should have been excluded but you didnt notice it. The counter simply displays what you want.
(I. S dont know if I explained that clearly, let me know ^^)

[mbroedl]

Mhmmm... do all the things you want to exclude have some kind of grammar structure in common that it'd be easy to exclude them?

For Markdown (and other grammars I am aware of) there is no such thing as: 'this is actual text without any fuss', but the whole document carries the same label, even if it's links, markup, etc.
So at least in the case of markdown it would actually be impossible to create an inclusion list.
And I guess usually they are written (and intended) to mark out things that are noteworthy, such as highlights, links, headings, etc.

Is the whole fountain document tokenised in depth? (You can check by opening the drop down console and checking the output of 'log cursor scope' at any given position and see whether the output differs between dialogue text, description, etc.) When you do that in a markdown document, you just have a main category 'text.md' that accounts for every single character in the document. (see the example below)

I'll give you an example.
The file.

# Hello
This is an _example_.

Translates to this hierarchical syntax tree with the language-markdown package:

text.md
|— heading.markup.md, heading-1.md
     |— (#) punctuation.md
     |— (Hello) [no further tokens]
|— (This is an ) [no further tokens]
|— emphasis.italic.markup.md
     |— (_) punctuation.md
     |— (example) [no further tokens]
     |— (_) punctuation.md
|— (.) [no further tokens]

So to count all words in this case I would need to exclude all punctuation below the document level (text.md).
To count everything that is in italics, I need to only count emphasis.italic.markup.md but may still want to exclude punctuation.md... in this case there would be no way to do this only via exclusion, because for example "This is an " does not have any tokens that the italics part does not have...

So although apart from your example (and the hypothetical example I made) I don't think there is many cases for requiring a certain grammar to be counted.
But I start to think it's reasonable to have the option (for whatever use case).
I mean programatically it would not be any real extra effort to both include and exclude.
But only listing inclusion might create issues in your case as well (as explained above), because for example the punctuation.* (markers for bold, italics, etc) would be tokens on a level below the dialogue. So they would be included if you just did inclusion.

Looks like we're starting to understand each other!

[mbroedl]

I just realised that if you go via inclusion you could only ever count words in your fountain file (assuming other grammars do not have dialogue.* markers, for example)?
Would that be acceptable?

[X-Raym]

@mbroedl I see the inclusion problem. Not sure how to tackle this.

Here is scope log on my Fountain file, with dialog name, dialog text, and description, (things I want in my count):

entity.name.character.fountain
string.other.dialogue.fountain
meta.paragraph.text

It seems indeed to have generic text too.

[mbroedl]

We could possibly just pass on whatever is in the text field to the scope selector?
https://github.com/atom/first-mate/blob/master/spec/scope-selector-spec.coffee#L33

So I guess in your case you could have (haven't trialled it):
dialogue | character | paragraph
if you don't care about any other wordcount, whereas if you would like to count also in Markdown then possibly with
(-fountain & *) | (fountain & ( ... )

Forwarding the text field into the scope selector would certainly be most powerful, but might be harder to debug for the user (cannot find a good scope selector documentation), and also it would remove any kind of possibility to manage that conveniently in a context menu (if users would apply complex patterns, that is).
Maybe one could use the , to delineate blocks to turn on and off? (Although commas and disjunctions (OR -> |) are both allowed in grouped brackets, but that should be manageable?)

EDIT: When using a comma with a positive (inclusive) match, then in my case for excluding comments AND quotes it'd need to be -comment & -quote, whereas with a negative (exclusive) match it would be easier: comment, quote and therefore might be better in a UI sense? Then I assume you would need to wrap your statement into a -(...) and it should all be good?
I tend towards going towards an exclusive option because this is likely to be the use case for most people?