[IDEA] a better way to highlight duplicates in a list

[Paulgilbert2000]

Request to have a java script macro, that returns a list of duplicates, or a data-tiddler with the duplicates as index and the color as a value.

https://groups.google.com/g/tiddlywiki/c/ahbcnXqVgl8

[Jermolene]

Thanks @Paulgilbert2000 but I think @AnthonyMuscio's solution answers the request adequately, I don't see the need for a new core primitive.

[pmario]

@Jermolene .. You may move this issue to the "discussions" section. .. I didn't think about it, as I did suggest to raise an issue here.

... In the thread I did suggest to raise an issue, because I thought, that this is a sensible desire, since we have the possibility to show duplicates in lists. ... But I was thinking a bit further.

eg: What if there are 6 tiddlers. A & B have field: x .. so they are "red". C & D have field: y .. so they are "red" and E &F have field: z .. so they are "red" ... --> IMO that's a problem. All elements are red now. Since all of them are red, there is no useful information anymore. ...

I would like to discuss a possibility to solve this problem with a decent amount of code. Tony's solution will break down with my testcase. ...

The other testcase I was thinking about was: I want to know the number of elements that are duplicated. Eg: field : x is there 3 times. ... and so on.

I think there is a reason for the count[] operator. I think there may be a reason for a duplicates macro too. ... BUT it may be different to count[]

[pmario]

I was thinking about a macro.

eg: <<duplicates filter="[<filter expression>]" mode="list / lookup-colour / lookup-counter" colours="colour-list" lookup-title="tiddler title">> ... or something similar.

[pmario]

<<duplicates filter="[<filter expression>]" mode="list / lookup-colour / lookup-counter" colours="colour-list" lookup-title="tiddler title">>

mode

• list ... The macro directly returns a list of results, which can be used using enlist-input[]
• lookup-colour ... The macro reads the "colour-list" from the "colours" parameter and assigns them to a data-tiddler. key : value -> result : colour
• lookup-counter ... The macro assigns the number of duplicates to a lookup tiddler. key : value -> result : number

colours

• space separated list of colours eg: red green blue black ... fallback colour may be "red" if not defined or out of possibilities

lookup-title

• title of a data-tiddler that contains the lookup-table.

[pmario]

IMO the advantage of the macro here would be, that we can do all the logic in 1 loop instead of many.

[Jermolene]

That might be an interesting macro to experiment with, but really doesn't seem like something for the core at this point in time.

[pmario]

You are right. That's why a discussion is the right place to think about it.

[pmario]

The second problem I have with the solution is: T(n) = O(n^2) ... Tony's solution has a worse performance characteristic, but may be optimized to fit O(n^2) .. But I think that's the best we can get.

With a "lookup" data tiddler, which I did suggest it should be possible to get O(2) in wikitext and O(n) in javascript ... So we may end up with O(n) + 2

[AnthonyMuscio]

Mario,

Although I answered the thread in TiddlyWiki forums with a method may call I may call "re-evaluate" on the whole list again is was already an edge case because a simple list sorted by the values in question would only need comparison with the last value to test for duplicates.

As is so often the case the "best" method is often related to the data complexity or volume, or volume of duplicates. In the example case they wanted it in title order, and they wanted the all primary and duplicates highlighted.

The requirement for a duplicates test is part of a suite of methods or features that have being neglected given the power of the list widget is already large. I have felt actually perhaps we need a "report widget" which serves normal report generation, including report and page heading and footers, access to last value, allow support for breaking on multiple fields/values eg; Country, state, region, city ie you can detect last city thus change in city to display city totals.

Your argument for a duplicates operator, is in my view more to allow logic simplification, rather than performance improvements even if it would be "faster", however there are other ways to solve the same problem.

Working through the duplicates question I think a solution that would provide a more general solution would be if the list widget had an additional variable passed to the "item template" and that is "previous title". This should have a very low performance impact, possibly close to none, as no doubt the last value is already available inside the widget code.

<$list filter="blah ordered by fieldname" last-title=last-title>
<$set name=lastvarvalue value={{{ [<last-title>get[fieldname]] }}}>
<$link/> {{{ [all[current]get[fieldname]match<lastvarvalue>then[duplicate]] }}}<br>
</$set>
</$list>

The above would only highlight the second or subsequent "duplicates".

• We can do a version of this already using the list widgets new (5.2.0) counter parameter as an index into any tiddler in a list with the $set widget select parameter (-1/-2) to recover the last title.
• Another way is to use filter on the set widget to generate a list first then process this inside a list widget.
• However a report widget could be built on top of the list widget but includes the tools for managing hierarchical lists.

What do you think, we could address a number of missing features in a report widget rather than continuing to load the list widget? I could start a thread on this?

Tony

[AnthonyMuscio]

We could first have a macro as a custom report tool.

[pmario]

@AnthonyMuscio Have you seen this discussion? #5674 (comment)

[pmario]

The <$log /> and <$action-log /> widget would already provide a possibility to save "report data" into a temporary memory.
It may be possible to read them with the proposed widget from #5674 ... So the data can be exposed to users.

[AnthonyMuscio]

Mario I would always support bringing more information to designers and users, however in this case with a report widget (or otherwise) I am thinking more of front end tool, which may benefit from such analytics, however the greater value would be providing a common algorithm rather than demanding people construct if from the pieces. In some ways it also goes to filling the gaps in TW5 that were solved in TWC with the ForEachTiddler Plugin.