From db4249412d2502794fdcd0186cd74a40f18efa34 Mon Sep 17 00:00:00 2001 From: Frames Catherine White Date: Tue, 20 Sep 2022 16:26:28 +0100 Subject: [PATCH 1/6] Comment on that all nonlinear operators can be shifted to augmented primal --- docs/src/design/changing_the_primal.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/src/design/changing_the_primal.md b/docs/src/design/changing_the_primal.md index 7c28fd464..a53814728 100644 --- a/docs/src/design/changing_the_primal.md +++ b/docs/src/design/changing_the_primal.md @@ -469,6 +469,15 @@ We don't have this in ChainRules.jl yet, because Julia is missing some definitio We have been promised them for Julia v1.7 though. You can see what the code would look like in [PR #302](https://github.com/JuliaDiff/ChainRules.jl/pull/302). +## What things can be pulled out of the pullback? +At this point you might wonder, is there a rule for what can be taken out of the pullback and computed in the augmented primal? +We can deduce one, or infact two. +The first and most practical one is that any computation that depends only on the primal input (or consequently its output) can be shifted to the augmented primal. +The second and perhaps more insightful is that all nonlinear parts can moved out (this is a weaker statement but more interesting). +We know this because pullbacks are linear operators -- they are linear in relation to the tangent they are pulling back. +This means they are in turn composed only of functions that linear operators (in relation to the tangent). +The fully minimized pullback function only calls linear operators -- the apparently nonlinear parts can all be shifted to the augmented primal. + ## Conclusion This document has explained why [`rrule`](@ref) is the way it is. In particular it has highlighted why the primal computation is able to be changed from simply calling the function. From 60c5e60322ad158669745746adaf02054cfe43e8 Mon Sep 17 00:00:00 2001 From: Frames Catherine White Date: Thu, 8 Dec 2022 21:11:10 +0000 Subject: [PATCH 2/6] Update docs/src/design/changing_the_primal.md --- docs/src/design/changing_the_primal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/design/changing_the_primal.md b/docs/src/design/changing_the_primal.md index a53814728..bd8cc99fd 100644 --- a/docs/src/design/changing_the_primal.md +++ b/docs/src/design/changing_the_primal.md @@ -476,7 +476,7 @@ The first and most practical one is that any computation that depends only on th The second and perhaps more insightful is that all nonlinear parts can moved out (this is a weaker statement but more interesting). We know this because pullbacks are linear operators -- they are linear in relation to the tangent they are pulling back. This means they are in turn composed only of functions that linear operators (in relation to the tangent). -The fully minimized pullback function only calls linear operators -- the apparently nonlinear parts can all be shifted to the augmented primal. +The minimal pullback function only calls linear operators -- the nonlinear parts can all be shifted to the augmented primal. ## Conclusion This document has explained why [`rrule`](@ref) is the way it is. From 41ee9af2966f81dcc6c9b0f22c903f7fed5b8b58 Mon Sep 17 00:00:00 2001 From: Frames Catherine White Date: Thu, 8 Dec 2022 21:11:18 +0000 Subject: [PATCH 3/6] Update docs/src/design/changing_the_primal.md Co-authored-by: Miha Zgubic --- docs/src/design/changing_the_primal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/design/changing_the_primal.md b/docs/src/design/changing_the_primal.md index bd8cc99fd..0f4b01758 100644 --- a/docs/src/design/changing_the_primal.md +++ b/docs/src/design/changing_the_primal.md @@ -475,7 +475,7 @@ We can deduce one, or infact two. The first and most practical one is that any computation that depends only on the primal input (or consequently its output) can be shifted to the augmented primal. The second and perhaps more insightful is that all nonlinear parts can moved out (this is a weaker statement but more interesting). We know this because pullbacks are linear operators -- they are linear in relation to the tangent they are pulling back. -This means they are in turn composed only of functions that linear operators (in relation to the tangent). +This means they are in turn composed only of functions that are linear operators w.r.t. the tangent. The minimal pullback function only calls linear operators -- the nonlinear parts can all be shifted to the augmented primal. ## Conclusion From 00a7a78831fdc8ded58a4b4b7938e86c501de4cb Mon Sep 17 00:00:00 2001 From: Frames Catherine White Date: Thu, 8 Dec 2022 21:16:33 +0000 Subject: [PATCH 4/6] Apply suggestions from code review Co-authored-by: Miha Zgubic --- docs/src/design/changing_the_primal.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/src/design/changing_the_primal.md b/docs/src/design/changing_the_primal.md index 0f4b01758..ce0064f35 100644 --- a/docs/src/design/changing_the_primal.md +++ b/docs/src/design/changing_the_primal.md @@ -472,9 +472,10 @@ You can see what the code would look like in [PR #302](https://github.com/JuliaD ## What things can be pulled out of the pullback? At this point you might wonder, is there a rule for what can be taken out of the pullback and computed in the augmented primal? We can deduce one, or infact two. -The first and most practical one is that any computation that depends only on the primal input (or consequently its output) can be shifted to the augmented primal. -The second and perhaps more insightful is that all nonlinear parts can moved out (this is a weaker statement but more interesting). -We know this because pullbacks are linear operators -- they are linear in relation to the tangent they are pulling back. +The first and most practical one is that any computation that depends only on the primal input or output can be shifted to the augmented primal. +The second, perhaps more insightful, rule is that all nonlinear parts can be moved out. +This is a weaker but more interesting statement. +We know this because pullbacks are linear operators -- linear w.r.t. the tangent they are pulling back. This means they are in turn composed only of functions that are linear operators w.r.t. the tangent. The minimal pullback function only calls linear operators -- the nonlinear parts can all be shifted to the augmented primal. From 028516b0a17a6786c290ec811c06e750996b936d Mon Sep 17 00:00:00 2001 From: Frames Catherine White Date: Thu, 8 Dec 2022 21:17:40 +0000 Subject: [PATCH 5/6] Update docs/src/design/changing_the_primal.md Co-authored-by: Miha Zgubic --- docs/src/design/changing_the_primal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/design/changing_the_primal.md b/docs/src/design/changing_the_primal.md index ce0064f35..cdcd3f51f 100644 --- a/docs/src/design/changing_the_primal.md +++ b/docs/src/design/changing_the_primal.md @@ -469,7 +469,7 @@ We don't have this in ChainRules.jl yet, because Julia is missing some definitio We have been promised them for Julia v1.7 though. You can see what the code would look like in [PR #302](https://github.com/JuliaDiff/ChainRules.jl/pull/302). -## What things can be pulled out of the pullback? +## What things can be taken out of the pullback? At this point you might wonder, is there a rule for what can be taken out of the pullback and computed in the augmented primal? We can deduce one, or infact two. The first and most practical one is that any computation that depends only on the primal input or output can be shifted to the augmented primal. From 7e63c2774765bc922e8b30e2d5b91e57cb3fcf30 Mon Sep 17 00:00:00 2001 From: Frames White Date: Mon, 27 Feb 2023 15:29:43 +0800 Subject: [PATCH 6/6] Update docs/src/design/changing_the_primal.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Mathieu Besançon --- docs/src/design/changing_the_primal.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/src/design/changing_the_primal.md b/docs/src/design/changing_the_primal.md index cdcd3f51f..4e72d0e96 100644 --- a/docs/src/design/changing_the_primal.md +++ b/docs/src/design/changing_the_primal.md @@ -471,7 +471,7 @@ You can see what the code would look like in [PR #302](https://github.com/JuliaD ## What things can be taken out of the pullback? At this point you might wonder, is there a rule for what can be taken out of the pullback and computed in the augmented primal? -We can deduce one, or infact two. +We can deduce one, or in fact two. The first and most practical one is that any computation that depends only on the primal input or output can be shifted to the augmented primal. The second, perhaps more insightful, rule is that all nonlinear parts can be moved out. This is a weaker but more interesting statement.