Early stopping

[oxinabox]

Consider a function foo(x) = f(g(h(x))) that we might want to take the derivative of.
If in this function the deriviative of g is zero, either in dynamic iszero sense,
or in the ChainRules stong Zero sense.
then we should not calculate the derivative of f.
We should just run f like a normal function.
since the pushforward is always linear,
so the deriviative of foo is going to be zero.

More generally for multiargument functions, need to check all arguments.

This can potentially give signficiant speedup,
since the basic implementation of f could be much simpler or at least more cache friendly.

This discussion comes out of:
JuliaDiff/ChainRulesCore.jl#90

Might take some redesign to make this work out.
but could be quite worth-wild.

[ChrisRackauckas]

Seems like that's a job for constant prop?

[oxinabox]

No, constant-prop can't be expected to do this.

Consider your example from: JuliaDiff/ChainRulesCore.jl#74

function frule(::typeof(concrete_solve),p,dp,u0,du0,odeprob)
  S = du0
  _prob = build_bigger_ode(odeprob,[u0,S])
  sol = solve(_prob,solver)
  y,dy = split_solution(sol)
  y,dy.*dp # weigh by the direction vector!
end

Its unreasonable to expect that constant propagation with iszero(dp) and iszero(du0)
to result in this not calling build_bigger_ode and solve.
And those are probably fairly expensive operations.

[ChrisRackauckas]

In this expression we'd manually do a check for iszero(du0), since it's a different algorithm if that's non-zero. That would probably still constant prop get rid of the branching though, not that it would really matter all that much in the grand scheme of things.

I don't think you'd ever get both being zero though?

[oxinabox]

I don't think you'd ever get both being zero though?

In this particular case either being zero is enough.
Thats not a generalizable property though.

I suspect there exists a suprising number of functions that meet the the requirement of not directly removable by constant prop (e.g. they call a function that will not constant prop out).
And that the user would not always want to insert the if iszero into the code themself.

But maybe there aren't.

One advantage of manuaully inserting the if iszero is that one can do cases like the above where eitehr argument being zero is enough.

---

Also neither of:

foo() = iszero([0.0, 0.0])
bar() = ([0.0, 0.0] .* [1.0,2.0])

Constant fold out.

[YingboMa]

No, Zero doesn't imply that we can stop. Consider

function foo(x)
    y = zero(eltype(x))
    for z in x
        y += z
    end
    return y
end

y would be Dual(0, [0, 0, 0, 0]), and we need [0, 0, 0, 0] to accumulate the differentiation result.

[oxinabox]

Doing the full forward mode AD just to get shape information seems sub-optimal.

We should be able to work out the shape information based on the shape of the primal and how many partials are being propagated.

Or we could work out a way to not care about the shape in the first place, when we know is just "all the zero you need".

[YingboMa]

Could you give an example where forming zeros can actually be avoided?

[oxinabox]

Could you give an example where forming zeros can actually be avoided?

Its an implementation detail. Kind of only related to this issue,
if its fundermentally impossible to solve -- that running AD is actually the only way to work out some information you need.

I am going to make a PR after we get this release out the door, to see if we can make dealing with zeros work.
Which is seperate from the issue of early stopping

[YingboMa]

I don't think that is an implementation detail. #18 (comment) you need zeros to start to accumulate.

[YingboMa]

Plus, since it is the forward mode, you don't see the computation ahead of you, so you cannot know when to stop.

[oxinabox]

Plus, since it is the forward mode, you don't see the computation ahead of you, so you cannot know when to stop.

You know exactly 1 step ahead of you.
Which is all you need.

[oxinabox]

Reopenning this, because it will be fixed by #20
Ergo it should be open so #20 can close it.

In particular it is closed by this line:
955e3ae#diff-1637e697eeadb4e03c76112b590f8077R155
since the value stops being a dual, which is how ForwardDiff2 knows not to do AD.

Arguably that line might want to use iszero so it catches dynamic zeros also.
But that would add overhead that is likely not worth it.