See if we can make Zero stronger again

[oxinabox]

It would be nice to be able to undo #90, and make Zero() * x = Zero() again.

This issue is to workout how.

[willtebbutt]

It seems like there was quite a lot of confusion in #90 regarding the design decision motivating retaining a very general strong Zero type in ChainRules, and the needs of ForwardDiff to have a "sensibly-defined" zero. I'm very keen to resolve this as soon as is humanly possible so that we're all in agreement and can move forward, so I've outlined my understanding of everything below.

Motivation for Zero

ChainRules's Differentials are very simply objects. You are only guaranteed to be able to do one thing with them: add them to other "conformal" differentials -- it's our contention that this is all that you really need for AD. (FWIW, it's been very illuminating to realise that this is really all that you need to be able to do).

As a consequence, there are some notable things you can only do some of the time:

• add differentials to their corresponding primal type
• convert differentials to a non-ChainRulesCore type

The former is not well-defined when constructors enforce invariants that adding a differential would violate. This is never an issue with Zero, since it literally changes nothing. Practically speaking, it's well defined most of the time, so we won't discuss it any further here.

The latter is particularly relevant here. Clearly Zero doesn't contain enough information to define convert(SomeType, Zero()) if the zero element depends on runtime information such as the size of an Array. The point is that this doesn't matter for AD - if you see a Zero, you know what it means the context of your frule / rrule by virtue of having access to a particular primal. It's essentially a way to inject compile-time information that says "don't worry about computing anything differential-related here, there's nothing to do".

The motivation for the very generic Zero we have here is

• it's always clear how to add it to other differentials
• it's always clear how to add it to primals
• it's a helpful optimisation in lots of situations -- accumulation in the reverse bit of reverse-mode, entirely avoiding forwards- or reverse- computations if the input to an frule / rrule is zero for some reason. There are probably things that I've not covered.

ForwardDiff2 motivation for converting to something like a differential

I don't have a good understanding of the internals of ForwardDiff2 -- as such this comment is a bit indecipherable to me. If either @ChrisRackauckas or @YingboMa could explain in more depth what's going on here, and more generally what the scope of your requirements are (both that are and aren't met by the current interface), that would be grand.

Relationship to #92

Whatever solution we come up with in #92 will necessary have to play nicely with our notion of a strong zero. Obviously we can propagate type information through straightforwardly here.

[oxinabox]

I am going to wait til we get the ForwardDiff2 initial release done, and then see about making a PR that makes it able to handle Zero under the old definition.

[YingboMa]

Sounds good, I am going to register ForwardDiff2 today, hopefully. :P Then, we can have more meaningful discussions about it.

[YingboMa]

Fixed by #99