Skip to content

Commit 1952dde

Browse files
committed
add instructions for performing crates.io follower maintenance
1 parent bef3a03 commit 1952dde

File tree

1 file changed

+74
-7
lines changed

1 file changed

+74
-7
lines changed

src/crates-io/db-maintenance.md

Lines changed: 74 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ can use:
2828
> visiting the website will still work, but logging in, publishing crates,
2929
> yanking crates or changing owners will not work.
3030
31-
## Primary Database Checklist
31+
## Primary database checklist
3232

3333
**1 hour before the maintenance**
3434

@@ -67,15 +67,24 @@ can use:
6767
same does not apply to the follower database, and there could be brief
6868
periods while the follower is not available.
6969

70-
3. Confirm the application is in read-only mode by trying to publish a crate
71-
and logging in.
70+
3. Wait for the application to be redeployed with the new configuration:
71+
72+
```
73+
heroku ps:wait -a crates-io
74+
```
7275
7376
3. Run the database maintenance:
7477
7578
```
7679
heroku pg:maintenance:run --force -a crates-io
7780
```
7881
82+
1. Wait for the maintenance to finish:
83+
84+
```
85+
heroku pg:wait -a crates-io
86+
```
87+
7988
3. Confirm all the databases are online:
8089
8190
```
@@ -99,8 +108,11 @@ can use:
99108
variable was *successfully removed*, it doesn't mean removing the variable
100109
failed. Failures are indicated with a red badge with a "x" (cross) in it.
101110
102-
3. Confirm the application is working by trying to publish a crate and logging
103-
in.
111+
3. Wait for the application to be redeployed with the new configuration:
112+
113+
```
114+
heroku ps:wait -a crates-io
115+
```
104116
105117
3. Update the status page and mark the maintenance as completed with this
106118
message:
@@ -132,5 +144,60 @@ can use:
132144
133145
# Follower database
134146
135-
Instructions and checklists for follower database maintenace aren't written
136-
yet.
147+
Performing maintenance on the follower database doesn’t require any external
148+
communication nor putting the application in read-only mode, as we can just
149+
redirect all of the follower’s traffic to the primary database. It shouldn’t be
150+
done during peak traffic periods though, as we’ll increase the primary database
151+
load by doing this.
152+
153+
## Follower database checklist
154+
155+
**At the start of the maintenance**
156+
157+
1. Configure the application to operate without the follower:
158+
159+
```
160+
heroku config:set -a crates-io DB_OFFLINE=follower
161+
```
162+
163+
1. Wait for the application to be redeployed with the new configuration:
164+
165+
```
166+
heroku ps:wait -a crates-io
167+
```
168+
169+
1. Start the database maintenance:
170+
171+
```
172+
heroku pg:maintenance:run --force -a crates-io READ_ONLY_REPLICA
173+
```
174+
175+
1. Wait for the maintenance to finish:
176+
177+
```
178+
heroku pg:wait -a crates-io READ_ONLY_REPLICA
179+
```
180+
181+
1. Confirm the follower database is ready:
182+
183+
```
184+
heroku pg:info -a crates-io
185+
```
186+
187+
1. Confirm the follower database is responding to queries:
188+
189+
```
190+
echo "SELECT 1;" | heroku pg:psql -a crates-io READ_ONLY_REPLICA
191+
```
192+
193+
1. Enable connections to the follower:
194+
195+
```
196+
heroku config:unset -a crates-io DB_OFFLINE
197+
```
198+
199+
1. Wait for the application to be redeployed with the new configuration.
200+
201+
```
202+
heroku ps:wait -a crates-io
203+
```

0 commit comments

Comments
 (0)