FilterDuplicate Operator

The FilterDuplicate operator can be leveraged to filter out duplicate expressions from an input source.

What	Type
Syntax	FilterDuplicate(expression)
expression	string
Return type	bool

First, the expression is evaluated to a string; the operator keeps a map of "seen" values and compares the expression string with what it already knows.

• Case a) The operator has not yet seen the value: In this case, FilterDuplicate returns false, but stores the value for future reference
• Case b) The operator has already seen the value
  • Case b1) The value is seen for exactly the second time: FilterDuplicate returns true exactly this time, and stores the information that this has been done
  • Case b2) The value is seen subsequent times: FilterDuplicate returns false again.

The FilterDuplicate operator should only be used inside a <SourceFilter> tag (see Config File Documentation).

Attention: Please note that this only filters for duplicate expressions; this does not mean that all other fields which you may or may not output in your field mapping also match among all records where the expression match. The values of the other fields you output will be taken from the record current in Case b1) above. This means, it is not a 100% equivalent of SQLs SELECT DISTINCT GROUP BY HAVING COUNT>1 feature.

Examples:

Filter	Description
<SourceFilter>FilterDuplicate($LastName)</SourceFilter>	Filters for duplicate $LastName
<SourceFilter>FilterDuplicate($FirstName + " " + $LastName</SourceFilter>	Filters for duplicate full names

See also FilterOnce Operator.