Description
Currently, test_frule is pretty useless if one of the partial derivatives is a NotImplemented since the derivative will always be a NotImplemented and hence tests always pass (even though marked as broken), regardless of the other derivatives. With JuliaDiff/ChainRulesCore.jl#477 it is possible to test the other partial derivatives by setting the tangents of the arguments for which the partial derivative is a NotImplemented to NoTangent() to test these partial derivatives properly. However, currently one has to do this manually, as in https://github.com/JuliaMath/SpecialFunctions.jl/blob/0af956882245e3b07340002c7c95c319e51af52a/test/chainrules.jl#L56-L57. It would be nice if these proper tests would be performed automatically.
Since it can't be inferred automatically which argument caused the NotImplemented derivative one approach might be to force users to always specify a NoTangent() tangent in the frule tests if the partial derivative is NotImplemented (if this is not already the default rand_tangent, in which case the partial derivative probably should just be NoTangent()), and basically not allow NotImplemented derivatives in the tests. A problem might be that this hides the fact that the implementation is broken and should be fixed in the frule tests. Maybe it would be better to let users specify a tangent of type NotImplemented (e.g. with @not_implemented()) and then use a NoTangent() internally but mark the tests as broken?
Originally posted by @devmotion in JuliaDiff/ChainRulesCore.jl#477 (comment)