Skip to content

Commit f9f1620

Browse files
freshstrangemusicfreshstrangemusic
committed
Update pref guidance
This patch overhauls the desktop pref experiment page with guidance on how to run pref setting experiments on desktop, with advice on patterns to avoid.
1 parent c9e281b commit f9f1620

File tree

1 file changed

+231
-38
lines changed

1 file changed

+231
-38
lines changed

‎docs/deep-dives/desktop/desktop-pref-experiments.md

Lines changed: 231 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,9 @@
11
---
22
id: desktop-pref-experiments
3-
title: Running Pref-setting Experiments on Desktop
3+
title: Desktop Pref Experiments
44
slug: /desktop-pref-experiments
55
---
66

7-
[test][prefFlips]
8-
97
As of Firefox 107, Nimbus supports experiments that set preferences on Desktop.
108
Unlike Normandy, Nimbus cannot set arbitrary preferences; instead, the
119
preferences that may be set are determined by the feature manifest.
@@ -15,32 +13,173 @@ Each variable in a Nimbus feature can set a single pref of any type.
1513
NB: Support for JSON variables was added in Firefox 126. The value of the pref
1614
will be `JSON.stringify(value)`.
1715

18-
## Example Feature
16+
::::caution Danger, Will Robinson!
17+
Prefs are complicated and it is easy to shoot yourself in the foot when using them if
18+
you're not cautious. The Nimbus team has prepared some guidance about using
19+
prefs in your feature:
20+
21+
If you do not actually *need* to use a pref in your feature, we recommend that
22+
you do not use one and instead read directly from the Nimbus API (via
23+
`NimbusFeatures.myFeature.getVariable("var")` /
24+
`NimbusFeatures.myFeature.getAllVariables()`).
25+
26+
If you *must* use a pref, then you must be careful about all writers to that
27+
pref. Please read [this section](#user-preference-changes) for a detailed
28+
explanation of
29+
pref writing interactions in Nimbus. We recommend that no other code write to
30+
that pref.
31+
32+
Finally, we advise against instructing users to edit the pref via
33+
`about:config`. This has a knock-on effect of normalizing users interacting with
34+
complex "under the hood" settings which can be taken advantage of by malicious
35+
actors. In addition, manually changing some prefs that are otherwise managed by
36+
client code can violate invariants and force clients into confusing or
37+
unpredicted scnearios.
38+
39+
If you've read to the end of this and aren't scared off, please read [this
40+
section](#pref-branches) on which branch your feature should write to.
41+
::::
42+
43+
## Pref branches
44+
45+
Each variable using `setPref` must specify which branch will be written to. There are two branches,
46+
each with its trade-offs:
47+
48+
<dl>
49+
<dt>The default branch</dt>
50+
<dd>
51+
The default branch is where Firefox's default pref values come from.
52+
53+
This branch <em>is not</em> persisted to disk and so there will be a time
54+
period during startup before Nimbus has finished initializing where default
55+
pref values will not represent the current experiment state.
56+
57+
You may want to use this branch if your code distinguishes from user branch
58+
values versus default branch values.
59+
</dd>
60+
61+
<dt>The user branch</dt>
62+
<dd>
63+
The user branch is where user's configuration choices are stored (e.g., when
64+
changing settings via <code>about:preferences</code> or { }
65+
<code>about:config</code>).
66+
67+
This branch <em>is</em> persisted to disk and loaded early during startup,
68+
before Nimbus is initialized. Additionally, if a pref has a value on both the
69+
default and user branches the user branch value will take precedence (e.g.,
70+
<code>Services.prefs.getBoolPref("foo.bar.baz")</code> will attempt to read
71+
from the user branch first).
72+
</dd>
73+
</dl>
74+
75+
If your feature configuration needs to be available early at startup (e.g., so
76+
that Gecko internals can be initialized properly) you will have to use the user
77+
branch.
78+
79+
Nimbus startup is triggered asynchronously after the UI has been shown
80+
(technically after `sessionstore-windows-restored`) or 5 seconds after
81+
`browser-before-ui-startup` (whichever comes first). Therefore, if your feature
82+
controls whether or not some amount of UI is shown in the browser chrome (e.g.,
83+
whether a button shows on the toolbar or not), you likely will want to use the
84+
user branch.
85+
86+
:::caution Respecting User Choice
87+
It is important to remember that writing to prefs on the user branch can
88+
**overwrite user choices**.
89+
90+
Experimenter can automatically exclude users that have changed prefs controlled
91+
by your feature. To enable this behaviour, check the "Prevent enrollment if
92+
users have changed any prefs set by this experiment" checkbox on the Branches
93+
page of your experiment. If you use this feature, you likely want to enable
94+
"Sticky Enrollment" on the audience page as well to prevent unexpected
95+
unenrollments.
96+
97+
You may also want to use this feature if you are writing to the default pref
98+
branch and any user branch overrides would cause breakage result in incorrect
99+
analysis.
100+
:::
101+
102+
## Configuring Your Feature in Experimenter
103+
104+
If you are configuring an experiment you should include the same prefs **in
105+
every branch of your experiment**. This protects your experiment from being
106+
impacted by user pref changes across all branches.
107+
108+
For example, let's consider the following feature:
19109

20110
```yaml
21-
my-feature:
22-
description: A description of my feature
23-
owner: whoami@mozilla.com
24-
hasExposure: false
111+
feature:
25112
variables:
26113
enabled:
27-
description: A variable setting a boolean pref to enable a feature.
28114
type: boolean
29115
setPref:
116+
# Defaults to false
117+
pref: myFeature.enabled
30118
branch: user
31-
pref: my_feature.enabled
32-
name:
33-
description: A variable setting a string pref to determine some name.
34-
type: string
119+
120+
optionalFeatureEnabled:
121+
type: boolean
35122
setPref:
123+
# Defaults to false
124+
pref: myFeature.optional.enabled
36125
branch: user
37-
pref: my_feature.name
38-
count:
39-
description: A variable setting an integer pref to determine some count.
40-
type: int
41-
setPref:
42-
branch: default
43-
pref: my_feature.count
126+
```
127+
If we have an experiment with three branches:
128+
129+
1. **Control**
130+
```
131+
{}
132+
```
133+
134+
2. **Treatment A**
135+
```
136+
{
137+
"enabled": true
138+
}
139+
```
140+
141+
3. **Treatment B**
142+
```
143+
{
144+
"enabled": true,
145+
"optionalFeatureEnabled": true
146+
}
147+
```
148+
149+
Nimbus only registers pref listeners for variables controlled by the current
150+
branch. So if the user is enrolled in the **Control** branch Nimbus will not
151+
listen for any pref changes. In the **Treatment A** branch Nimbus will listen
152+
for changes to `myFeature.enabled`. In the **Treatment B** branch Nimbus will
153+
listen to changes to both `myFeature.enabled` and `myFeature.optional.enabled`.
154+
155+
It is therefore possible for a user to be in the **Control** branch with
156+
`myFeature.enabled` set to true and it is also possible for a user to tbe in
157+
**Treatment A** branch and have `myFeature.optional.enabled` set to true.
158+
159+
The proper way to set up this experiment is therefore as follows:
160+
161+
1. **Control**
162+
```
163+
{
164+
"enabled": false,
165+
"optionalFeatureEnabled": false
166+
}
167+
```
168+
169+
2. **Treatment A**
170+
```
171+
{
172+
"enabled": true,
173+
"optionalFeatureEnabled": false
174+
}
175+
```
176+
177+
3. **Treatment B**
178+
```
179+
{
180+
"enabled": true,
181+
"optionalFeatureEnabled": true
182+
}
44183
```
45184

46185
## Experiments vs Rollouts
@@ -61,14 +200,40 @@ first enrollment, with some caveats:
61200
* If the pref is set on the user branch and the pref was not set before
62201
enrollment, then the pref will be cleared and will be no longer available.
63202

64-
## Pref branches
203+
## Unenrollment from Pref Experiments
204+
205+
Normally Nimbus will only unenroll from experiments and rollouts when we check
206+
for new recipes (periodically or after startup). During that check, Nimbus will
207+
unenroll if:
208+
209+
* the experiment or rollout is no longer present on Remote Settings (reported as
210+
`recipe-not-seen` in telemetry);
211+
* the experiment or rollout has a targeting expression that is no longer true
212+
(reported as `targeting-mismatch`);
213+
* the rollout no longer matches bucketing (reported `bucketing`).
214+
215+
:::caution Including prefs in targeting expressions
216+
If your experiment or rollout includes preference checks in its targeting
217+
expression then it may unexpectedly unenroll or prevent a rollout or experiment
218+
on the same feature from enrolling.
219+
220+
Experimenter can automatically exclude users that have changed prefs controlled
221+
by your feature. If your feature writes to the user branch and uses this feature
222+
(the "Prevent enrollment if users have changed any prefs set by this experiment"
223+
checkbox on the branches page), you **must** enable "sticky enrollment" on the
224+
Audience Page. Otherwise your experiment will automatically unenroll the next
225+
time Nimbus evaluates targeting.
226+
:::
65227

66-
Each variable using `setPref` must specify which branch will be written to.
67-
The default branch is not persisted, so prefs set on the default branch will not
68-
be available until Nimbus completes its startup and loads all its active
69-
experiments from disk.
228+
Pref Experiments can unenroll for additional reasons:
70229

71-
## User Preference Changes
230+
* the experiment or rollout sets a pref and that pref changes (either by a user
231+
making a change in `about:config` or client code using the pref API);
232+
* the feature manifest changed sufficiently; or
233+
* Nimbus enrolled in an [Incident Response Pref Flip][prefFlips] that set the
234+
same pref.
235+
236+
### Unexpected Preference Changes
72237

73238
If a user is enrolled in an experiment or rollout that sets a pref and that pref
74239
changes, the user will be unenrolled from the experiment (or rollout). This
@@ -78,7 +243,7 @@ are experimenting on, otherwise their populations may get spuriously unenrolled.
78243

79244
The new value of the preference will be persisted.
80245

81-
## Manifest Changes
246+
### Manifest Changes
82247

83248
Some changes to the feature manifest may result in unenrollment from an active
84249
experiment:
@@ -95,17 +260,7 @@ pref-setting and non-pref setting variables, then changes to the manifest will
95260
not result in unenrollment if the active experiment does not have any values for
96261
pref-setting variables.
97262

98-
## Restrictions with Fallback Prefs
99-
100-
Variables may not specify both a `fallbackPref` and a `setPref`.
101-
102-
Fallback prefs and set prefs are mutually exclusive. That is, If any variable in
103-
any feature specifies a pref as a fallback pref, no variable may set that
104-
variable as a set pref and vice versa.
105-
106-
These restrictions are enforced at build time.
107-
108-
## Conflicts with Incident Response Pref Flips
263+
### Conflicts with Incident Response Pref Flips
109264

110265
If a user is enrolled in a setPref experiment/rollout and then enrolls in an
111266
[incident response pref flip][prefFlips], they will be unenrolled from the
@@ -135,6 +290,44 @@ following data:
135290
</tbody>
136291
</table>
137292

293+
## Restrictions with Fallback Prefs
294+
295+
Variables may not specify both a `fallbackPref` and a `setPref`.
296+
297+
Fallback prefs and set prefs are mutually exclusive. That is, If any variable in
298+
any feature specifies a pref as a fallback pref, no variable may set that
299+
variable as a set pref and vice versa.
300+
301+
These restrictions are enforced at build time.
302+
303+
## Example Feature
304+
305+
```yaml
306+
my-feature:
307+
description: A description of my feature
308+
owner: whoami@mozilla.com
309+
hasExposure: false
310+
variables:
311+
enabled:
312+
description: A variable setting a boolean pref to enable a feature.
313+
type: boolean
314+
setPref:
315+
branch: user
316+
pref: my_feature.enabled
317+
name:
318+
description: A variable setting a string pref to determine some name.
319+
type: string
320+
setPref:
321+
branch: user
322+
pref: my_feature.name
323+
count:
324+
description: A variable setting an integer pref to determine some count.
325+
type: int
326+
setPref:
327+
branch: default
328+
pref: my_feature.count
329+
```
330+
138331
139332
[prefFlips]: /desktop-incident-response
140333
[glean-telemetry]: https://dictionary.telemetry.mozilla.org/apps/firefox_desktop/metrics/nimbus_events_unenrollment

0 commit comments

Comments
 (0)