Functions must begin with :or, or :all (=) to receive input

[btheado]

Based on experiments, I find that functions must begin with :or (or blank), or :all (or =) prefix in order to receive input. Given that constraint, functions which need to process input should using one of the other filter run prefixes should start always start with [all[]] , or :or[all[]], or :all[all[]], or =[all[]]. The all operator serves as an identity/no-op/passthrough operator. There are other operators which can act as passthrough (i.e. [prefix[]] with no argument), but [all[]] might be the most clear.

I did not find any mention of this requirement in the docs, so I plan to add it. But one thing I don't know is the reason for it. Does it have to be this way? I could just document the requirement as a fact, but it would be nice to include an explanation. Can anyone help with the explanation?

Here is a passing test case which shows any named prefix can receive input as long as it starts with [all[]]

title: Functions/FunctionNamedFilterRunPrefix1
description: Functions with named prefixes receive input using [all[]]
type: text/vnd.tiddlywiki-multiple
tags: [[$:/tags/wiki-test-spec]]

title: Output

\whitespace trim
\function all.identity()                  :all[all[]]
\function and.identity()          [all[]] :and[all[]]
\function cascade.identity()      [all[]] :cascade[all[]]
\function else.one()              [all[]] :else[[1]]
\function except.two()            [all[]] :except[[2]]
\function filter.identity()       [all[]] :filter[all[]]
\function intersection.identity() [all[]] :intersection[all[]]
\function map.identity()          [all[]] :map[all[]]
\function or.identity()                   :or[all[]]
\function reduce.identity()       [all[]] :reduce[all[]]
\function sort.identity()         [all[]] :sort[all[]]
\function then.identity()         [all[]] :then[all[]]

<$text text={{{ 
  [title[all.identity]all.identity[]]
  [title[and.identity]and.identity[]]
  [title[cascade.identity]cascade.identity[]]
  [title[else.one]else.one[]]
  [enlist[2 except.two]except.two[]]
  [title[filter.identity]filter.identity[]]
  [title[intersection.identity]intersection.identity[]]
  [title[map.identity]map.identity[]]
  [title[or.identity]or.identity[]]
  [title[reduce.identity]reduce.identity[]]
  [title[sort.identity]sort.identity[]]
  [title[then.identity]then.identity[]]
  :and[join[ ]]
}}}/>

+
title: ExpectedResult

all.identity and.identity cascade.identity else.one except.two filter.identity intersection.identity map.identity or.identity reduce.identity sort.identity then.identity

Here is a test case which shows without the [all[]], most of the prefixes don't receive input

title: Functions/FunctionNameFilterRunPrefix2
description: Only functions starting with :or or :all named prefixes receive input
type: text/vnd.tiddlywiki-multiple
tags: [[$:/tags/wiki-test-spec]]

title: Output

\whitespace trim
\function all.identity()          :all[all[]]
\function and.identity()          :and[all[]]
\function cascade.identity()      :cascade[all[]]
\function else.one()              :else[[1]]
\function except.two()            :except[[2]]
\function filter.identity()       :filter[all[]]
\function intersection.identity() :intersection[all[]]
\function map.identity()          :map[all[]]
\function or.identity()           :or[all[]]
\function reduce.identity()       :reduce[all[]]
\function sort.identity()         :sort[all[]]
\function then.identity()         :then[all[]]

<$text text={{{ 
  [title[all.identity]all.identity[]]
  [title[and.identity]and.identity[]]
  [title[cascade.identity]cascade.identity[]]
  [title[else.one]else.one[]]
  [enlist[2 except.two]except.two[]]
  [title[filter.identity]filter.identity[]]
  [title[intersection.identity]intersection.identity[]]
  [title[map.identity]map.identity[]]
  [title[or.identity]or.identity[]]
  [title[reduce.identity]reduce.identity[]]
  [title[sort.identity]sort.identity[]]
  [title[then.identity]then.identity[]]
  :and[join[ ]]
}}}/>

+
title: ExpectedResult

all.identity 1 or.identity

I wrote similar test cases for the subfilter operator and the behavior is consistent.

Check out a live version at the share site

Edit: Added explicit call to the title operator to hopefully make it a little more clear.

[pmario]

I am not sure what you want to show with your testcases.

There is the function-operator, which allows us to call functions in filter-runs

New in v5.3.0 The function operator applies a named function to the input titles, and returns the results from the function. The function is invoked once with all of the input titles (in contrast, the filter Operator invokes its function separately for each input title).

If you want to call a function with inputs, you have to give it input parameters

\function test-match(var1, var2) [<var1>match[one]else<var2>] 

{{{ [function[test-match],[one],[two]]  }}}

{{{ [function[test-match],[oneXX],[two]]  }}}

Calling functions without the function[] operator is possible, if you include a dot in the function name. We could call that a shortcut.

\function test.match(var1, var2) [<var1>match[one]else<var2>] 

{{{ [<test.match one two>]  }}}

{{{ [<test.match oneXX two>]  }}}

But that's probably not what you want to show us.

[btheado]

Thanks, @pmario. In trying to demonstrate every filter prefix, I see I made it confusing.

I'll try again.

\function map.identity1()          [all[]] :map[all[]]
\function map.identity2()                  :map[all[]]

<$text text={{{ [title[abc]map.identity1[]] }}}/> - <$text text={{{ [title[abc]map.identity2[]] }}}/>

My question is why is the output of the above abc - instead of abc - abc. Why isn't the input supplied to the map.identity2 function?

I see the same behavior for all the other filter run prefixes except for :all and :or.

[Jermolene]

Thanks @btheado, very intriguing. I would also think that [all[]] should be a no-op in this case. At the moment I'm baffled; the implementation of "[all[]]" is very simple, and it's really hard to see how this can be happening. I'll try coming back to it with fresh eyes.

I did create a simple test case:

title: Filters/FunctionInput
description: Verify expected inputs to functions
type: text/vnd.tiddlywiki-multiple
tags: [[$:/tags/wiki-test-spec]]

title: Output

\whitespace trim
\function map.identity1()          [all[]] :map[all[]]
\function map.identity2()                  :map[all[]]

<$text text={{{ [title[abc]map.identity1[]] }}}/> - <$text text={{{ [title[abc]map.identity2[]] }}}/>

<$text text={{{ [title[abc]function[map.identity1],[]] }}}/> - <$text text={{{ [title[abc]function[map.identity2],[]] }}}/>
+
title: ExpectedResult

<p>abc-abc</p><p>abc-abc</p>

[btheado]

the implementation of "[all[]]" is very simple

Right, it doesn't seem to have anything to do with the operator. You could use any "no-op" operator in its place such as [prefix[]]

The filter run prefix is what matters. Since [all[]] is an implicit :or and :or is one of the two filter prefixes for which it works. You can replace map.identity1 with

\function map.identity1()          :all[all[]] :map[all[]]

and it still gives the abc output.

[btheado]

Here is my analysis of the code. I don' t have a good big picture understanding of source and results, but the reason functions which begin with filter run prefixes such as :map don't see the input is because the results are not passed along to the function's "subfilter". When results start empty for these filter runs, the source is ignored. Filter run prefixes such as :all and :or don't care that the results start empty. The source will be processed just fine.

The function operator calls getVariableInfo, passing source in the options parameter:

TiddlyWiki5/core/modules/filters/function.js

Lines 24 to 26 in 028c807

	var variableInfo = options.widget && options.widget.getVariableInfo && options.widget.getVariableInfo(functionName,{params: params, source: source});
	if(variableInfo && variableInfo.srcVariable && variableInfo.srcVariable.isFunctionDefinition) {
	return variableInfo.resultList ? variableInfo.resultList : [variableInfo.text];

getVariableInfo calls this.wiki.filterTiddlers. The options.source is passed along:

TiddlyWiki5/core/modules/widgets/widget.js

Line 164 in 028c807

resultList = this.wiki.filterTiddlers(value,this.makeFakeWidgetWithVariables(variables),options.source);

Each filter run always starts with empty results due to this code:

TiddlyWiki5/core/modules/filters.js

Lines 344 to 349 in 028c807

	if(!widget) {
	widget = $tw.rootWidget;
	}
	var results = new $tw.utils.LinkedList();
	self.filterRecursionCount = (self.filterRecursionCount || 0) + 1;
	if(self.filterRecursionCount < MAX_FILTER_DEPTH) {

The :all filter run prefix (:or is similar) doesn't care the input results are empty...it will directly use source to get the input values. Therefore, these filter runs are able to see the input:

TiddlyWiki5/core/modules/filterrunprefixes/all.js

Lines 19 to 22 in 028c807

	exports.all = function(operationSubFunction) {
	return function(results,source,widget) {
	results.push.apply(results, operationSubFunction(source,widget));
	};

However, the :map filter run prefix and most of the other filter run prefixes explicitly return early when empty input results are encountered. Therefore these filter runs are NOT able to see the input:

TiddlyWiki5/core/modules/filterrunprefixes/map.js

Lines 15 to 19 in 028c807

	exports.map = function(operationSubFunction,options) {
	return function(results,source,widget) {
	if(results.length > 0) {
	var inputTitles = results.toArray(),
	index = 0,

The :and filter doesn't have a check for empty results, but it does ignore the input source and generates a new one from results.toArray(). Since since results is empty, this newly generated source doesn't give any output. This filter run is also unable to see the input:

TiddlyWiki5/core/modules/filterrunprefixes/and.js

Lines 19 to 24 in 028c807

	exports.and = function(operationSubFunction,options) {
	return function(results,source,widget) {
	// This replaces all the elements of the array, but keeps the actual array so that references to it are preserved
	source = options.wiki.makeTiddlerIterator(results.toArray());
	results.clear();
	results.pushTop(operationSubFunction(source,widget));

Edit: Link to the correct filter run results initialization

[Jermolene]

Thank you @btheado. So it seems like we have definite bugs in the map and and filter run prefixes; this would appear not to just be a documentation issue. It would be interesting to see if there was any unexpected impact if they were to be fixed.

[btheado]

I made this horrible, hacky change to automatically insert [all[]] at the start of any filter for which source is not empty and the first filter run is not a :all or :or filter run

diff --git a/core/modules/filters.js b/core/modules/filters.js
index aa82a352a..3cf3a0563 100644
--- a/core/modules/filters.js
+++ b/core/modules/filters.js
@@ -216,6 +216,15 @@ exports.getFilterRunPrefixes = function() {
 
 exports.filterTiddlers = function(filterString,widget,source) {
        var fn = this.compileFilter(filterString);
+       if (source) {
+               var filterParseTree = this.parseFilter(filterString);
+               if (filterParseTree.length > 0) {
+                       var firstRun = filterParseTree[0];
+                       if (firstRun.prefix != "" && firstRun.namedPrefix != "or" && firstRun.prefix != "=" && firstRun.namedPrefix != "all") {
+                               fn = this.compileFilter("[all[]]" + filterString);
+                       }
+               }
+       }
        return fn.call(this,source,widget);
 };

It would be interesting to see if there was any unexpected impact if they were to be fixed.

All tests pass, but only the tests I provided in OP actually exercise this new code.

It is hard for me to imagine someone having code which depends on something like \function map.identity() :map[all[]] returning empty results, but maybe I need to stretch my imagination :-).

I wonder if @saqimtiaz has any input here. I have a vague recollection that he recommended here or on the forums about inserting the [all[]] at the start of a function in order to get input fed to :map. I doubt I would have come up with this [all[]] workaround without that vague recollection. I can't seem to find reference to his recommendation, though.

I'm not sure what the "proper" fix corresponding to the above would look like. I need to learn more about the code in filters.js.

[Jermolene]

Linking to a discussion about the same issue: https://talk.tiddlywiki.org/t/wikitext-code-editor/11211

[btheado]

That discussion appears unrelated. Maybe you meant this one: https://talk.tiddlywiki.org/t/is-this-a-bug-cant-directly-use-prefixed-filter-run-in-function/11313

[Jermolene]

Thanks @btheado my mistake.

[btheado]

I found this code which seems to be a workaround that is somewhat related:

TiddlyWiki5/core/modules/widgets/action-listops.js

Lines 72 to 76 in 1f34c62

	if(this.subfilter) {
	var inputList = this.wiki.getTiddlerList(this.target,field,index),
	subfilter = "[all[]] " + this.subfilter;
	this.wiki.setText(this.target, field, index, $tw.utils.stringifyList(this.wiki.filterTiddlers(subfilter,this,this.wiki.makeTiddlerIterator(inputList))));
	}

It was introduced in #6540. @Jermolene, it looks like you added this.wiki.makeTiddlerIterator(inputList) as the source parameter to the this.wiki.filterTiddlers call. In the same commit you automatically prepend "[all[]] " to the subfilter.

Maybe you already knew [all[]] prefix would be required since most filter run prefixes ignore the source when the results linked list is empty. Or maybe you discovered that by trial and error. It is hard to tell from the history.

[Jermolene]

Hi @btheado I think this might be an accidental similarity, depending on how one looks at it. The approach taken by #6540 is to try to retain backwards compatibility with the existing code while avoiding using dangerous stringified list construction. The code looks like that because there is no API for passing a list of titles as the starting accumulator of a filter, which is of course the peculiarity we have been discussing.