[FEATURE] Use anti-joins for incremental updates

[nando-dirks]

Is your feature request related to a problem? Please describe.
Kind of, we have been noticing some queries slowing down on larger incremental tables, which seem to be running faster when using a anti-join instead of a left join + where statement in databricks

Describe the solution you'd like
Use anti-join instead of left join + where key is null

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

AB#5703

[nando-dirks]

Might already be resolved in 0.11

[nando-dirks]

checked on 0.11.2. An anti join is still significantly faster for our specific cases

[DVTimWilson]

Hi @nando-dirks, thank you for raising this issue. I am not a Databricks expert but I am guessing that using the Databricks' specific LEFT ANTI JOIN syntax causes the optimizer to skip the filtering step that the generic LEFT JOIN + WHERE syntax involves, so this will definitely be more performant at scale.

We shall investigate this further and consider including a Databricks specific revision in the next release.

Tim

cc @DVAlexHiggs