Ideas from "First-Class Automatic Differentiation in Swift: A Manifesto"

[datnamer]

https://gist.github.com/rxwei/30ba75ce092ab3b0dce4bde1fc2c9f1d

@jrevels this is a very very interesting read that has some potential ideas for Julia's AD ecosystem.

@jekbradbury knows more.

[jekbradbury]

There was lots of discussion of this and related things in the #autodiff channel today; perhaps it’s worth moving some of that to a more permanent place

[jrevels]

Yes, I have read it. Copy/pasting/editing some of my comments from Slack:

This manifesto aligns fairly well with my direction. I believe that Capstan can probably be a bit more ambitious in the long run, though, since Julia can exploit runtime type info in ways Swift currently cannot.

It seems like the Swift team will soon have the machinery in place to develop mixed-mode, but are not necessarily concerned with mixed-mode as a first-class goal right now. I think this is unfortunate, as
a mixed-mode approach can loosen restrictions on target language programmability without sacrificing as much performance as pure forward/reverse-mode approaches (since some subprograms that are intractable in one mode can be differentiated efficiently in the other).

I'm currently in the early stages of designing a markup language for specifying "differentiable signatures", defining mixed-mode chain rules, annotating functional properties like linearity etc. that will essentially be Capstan's version of DiffRules.jl. IMO, the primitive registration format laid out in the manifesto is quite interesting and highly related to this endeavor.