speed up field indexer by archetype ecs

[linonetwo]

Entity component system with archetype Can filter entity with very fast speed.
And those components are similar to tiddler fields, so maybe we can apply this (only to the indexer is sufficient) . This may also improve filters like tag filter.

I learned about this time ago. Just happen to think about this after dinner, may add more details later.

[pmario]

I did have a look, when the FieldIndexer actually is used. As far as I can see, it is only used once by the filters/field.js with wiki.eachXX.byField() function.

IMO we should start to use it instead of making it faster.

So basically if you hardcode eg: [code-body[yes]] it is used. But that's not a real world usecase, which needs to be fast.

IMO real world would be eg: [has<field-name>get<field-name>], where field-name=code-body. There it's not used at all.

[linonetwo]

Oh, I thought many filter operators was using indexer, so I come up with this.

So maybe let them using the indexer is another low-hanging fruit.

[pmario]

... So maybe let them using the indexer is another low-hanging fruit.

Yes.
As I wrote below. IMO has<field>,<value> may be a candidate.
It would be interesting to implement it and make some performance measurements, if it would be worth it.

[pmario]

eg: My tocP does this to "find all tiddlers, where the parent-field has the value of the __tag__-variable"

<$list filter="""[has<__field__>!has[draft.of]] -[<__tag__>] -[subfilter<__exclude__>] :filter[get<__field__>match<__tag__>] +[subfilter<__sort__>]""">

IMO it should be something similar to

<$list filter="""[has<__field__>,<__tag__>!has[draft.of]] -[<__tag__>] -[subfilter<__exclude__>] +[subfilter<__sort__>]""">

Which imo would significantly speed up the whole thing, because the initial filter-run [has<__field__>,<__tag__>!has[draft.of]] will be the result already.

@Jermolene ... Would it make sense to extend the has-operator with a second parameter, which would be the value. So IMO has could make use of the FieldIndexers. eg: [has<fieldName>,<value>]

[pmario]

There may be a other Filter Operators that could be improved. We would have to have a look, which one of them also deals with fields as their primary parameter.

I've long lost the overview, due to the amount (166) of operators we have atm, that do similar things. :/

[linonetwo]

I read the code of https://github.com/hmans/miniplex (less than 1000 loc)

I find ECS is like:

1. filter expressions: Query in ECS
2. add a new filter expressions: ECS will add a new bucket as cache, storing result of query. So multiple same query don't need to re-execute multiple times
3. run filter expression: instead of execute some JS on querying, it directly read data from cache.
4. change tiddler: in ECS, it will update each query bucket. Like update every filter expression.

So TW's filter expression is already simillar to ECS. Only difference is ECS will cache result of each query. This works when there are many same query. But we probably don't have many same filter expressions, so caching is useless.

So I'm moving my attention to:

#8618 (reply in thread)

[linonetwo]

ECS is about making tons of indexer, and re-run function that observing changed indexers.

TiddlyWiki is now re-run filter expression when any tiddler changes, and let widget itself decide re-run rendering or not, but filter is already run, we can't save filter's execution time.
A tiddler's content changes, Sidebar-Recent will re-run filter, and do expensive date based sorting from a full wiki, which is at least O(nlogn) because it is a sorting. [!is[system]$(subfilter)$has<dateField>!sort<dateField>limit<limit>]

With archetype ECS, or we can say it as "with tons of indexers", an archetype for filter in Sidebar-Recent will be !system + modified, and let it be created as sorted heap, its tiddler deletion + re-insertion will only cost O(logn) , so Sidebar-Recent's filter's sort part will be faster.

And archetype ECS also means remove indexer when no query is observing it. So in Tiddlywiki this will means when no UI is using that filter, it's related indexer is removed, and recreated when Recent tab is opened next time. What if user tend to switching between sidebar tabs frequently? Then tons of indexer will create and destroy frequently too.
And it requires analyse filter expression AST to know what indexer it needs to create, like getting !system + modified archetype indexer from !is[system] and sort<dateField> two parts.

So there is always a trade off, in a huge wiki, we can't guarantee archetype ECS can always provides boost.

[linonetwo]

The best way to speed up filter expressions may still be "throttle" and "requestIdleCallback".