Skip to content

Commit 26b2012

Browse files
committed
Move Ruby doc on db upgrades to common docs dir
And explain downgrades
1 parent ee5068d commit 26b2012

File tree

2 files changed

+135
-91
lines changed

2 files changed

+135
-91
lines changed

docs/prepare-db-upgrade.md

Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
# Upgrading a language database schema
2+
3+
When a `.dbscheme` file changes, you need to provide two things:
4+
5+
1. An upgrade script that modifies old databases (built against an earlier schema), so they can use new query functionality (albeit with possibly degraded results).
6+
2. A downgrade script that reverses those changes, so that newer databases can be queried using older query and library packs.
7+
8+
This document explains how to write or generate those scripts.
9+
10+
## Process Overview
11+
12+
1. Commit the change to your language's `.dbscheme` file, along with any library updates required to work with the change.
13+
2. Run `misc/scripts/prepare-db-upgrade.sh --lang <lang>`. This will generate skeleton upgrade/downgrade scripts in the appropriate directories.
14+
3. Fill in the details in the two `upgrade.properties` files that it generated, and add any required upgrade queries.
15+
16+
It may be helpful to look at some of the existing upgrade/downgrade scripts, to see how they work.
17+
18+
## Details
19+
20+
An `upgrade.properties` file will look something like:
21+
22+
```
23+
description: what it does
24+
compatibility: partial
25+
some_relation.rel: run some_relation.qlo
26+
```
27+
28+
The `description` field is a textual description of the aim of the upgrade.
29+
30+
The `compatibility` field takes one of four values:
31+
32+
* **full**: results from the upgraded snapshot will be identical to results from a snapshot built with the new version of the toolchain.
33+
34+
* **backwards**: the step is safe and preserves the meaning of the old database, but new features may not work correctly on the upgraded snapshot.
35+
36+
* **partial**: the step is safe and preserves the meaning of the old database, but you would get better results if you rebuilt the snapshot with the new version of the toolchain.
37+
38+
* **breaking**: the step is unsafe and will prevent certain queries from working.
39+
40+
The `some_relation.rel` line(s) are the actions required to perform the database upgrade. Do a diff on the the new vs old `.dbscheme` file to get an idea of what they have to achieve. Sometimes you won't need any upgrade commands – this happens when the dbscheme has changed in "cosmetic" ways, for example by adding/removing comments or changing union type relationships, but still retains the same on-disk format for all tables; the purpose of the upgrade script is then to document the fact that it's safe to replace the old dbscheme with the new one.
41+
42+
Ideally, your downgrade script will perfectly revert the changes applied by the upgrade script, such that applying the upgrade and then the downgrade will result in the same database you started with.
43+
44+
Some typical upgrade commands look like this:
45+
46+
```
47+
// Delete a relation that has been replaced in the new scheme
48+
obsolete.rel: delete
49+
50+
// Create a new version of a table by applying an expression (using a simple
51+
// synthetic language) to an existing table. The example duplicates the 'id'
52+
// column of input.rel as the last column of extended.rel, perhaps to record our
53+
// best guess at newly-populated "source declaration" information.
54+
extended.rel: reorder input.rel (int id, string name, int parent) id name parent id
55+
56+
// Create relationname.rel by running relationname.qlo and writing the query
57+
// results as a .rel file. The query file should be named relationname.ql and
58+
// should be placed in the upgrade directory. It should avoid using the default
59+
// QLL library, and will run in the context of the *old* dbscheme.
60+
relationname.rel: run relationname.qlo
61+
```
62+
63+
### Testing your scripts
64+
65+
Although we have some automated testing of the scripts (e.g. to test that you can upgrade databases all the way from the an initial dbscheme to the newest, and back), it's essential that you apply some more rigorous testing any non-trivial upgrade or downgrade. You might do so as follows:
66+
67+
#### Running qltests
68+
69+
To test the upgrade script, run:
70+
71+
```
72+
codeql test run --search-path=<old-extractor-pack>;ql <test-dir>
73+
```
74+
75+
Where `<old-extractor-pack>` is an extractor pack containing the old extractor and dbscheme that pre-date your changes, and `<test-dir>` is the directory containing the qltests for your language. This will run the tests using an old extractor, and the test databases will all be upgraded in place using your new upgrade script.
76+
77+
To test the downgrade script, create an extractor pack that includes your new dbscheme and extractor changes. Then checkout the `main` branch of `codeql` (i.e. a branch that does not include your changes), and run:
78+
79+
```
80+
codeql test run --search-path=<new-extractor-pack>;ql <test-dir>
81+
```
82+
83+
This will run the tests using your new extractor, and the databases will be downgraded using your new downgrade script so that they match the dbscheme of the `main` branch.
84+
85+
#### Manual testing
86+
87+
You might also choose to test with a real-world database.
88+
89+
1. Create a snapshot of your favourite project using the old version of the code.
90+
91+
2. Switch to the new version of the code.
92+
93+
3. Try to run some queries that will depend on your upgrade script working correctly.
94+
95+
4. Observe the upgrade being performed in the query server log.
96+
97+
5. Verify that your queries produced sensible results.
98+
99+
#### Doing the upgrade manually
100+
101+
To create the upgrade directory manually, without using `prepare-db-upgrade.sh`:
102+
103+
1. Get a hash of the old `.dbscheme` file from `main` (i.e. from just before your changes). You can do this by checking out the code prior to your changes and running `git hash-object ql/lib/<mylang>.dbscheme`
104+
105+
2. Go back to your branch and create an upgrade directory with that hash as its name, for example:
106+
```
107+
mkdir ql/lib/upgrades/454f1e15151422355049dc4f1f0486a03baeffef
108+
```
109+
110+
111+
3. Copy the old `.dbscheme` file to that directory, using the name old.dbscheme.
112+
113+
```
114+
cp ql/lib/<mylang>.dbscheme ql/lib/upgrades/454f1e15151422355049dc4f1f0486a03baeffef/old.dbscheme
115+
```
116+
117+
4. Put a copy of your new `.dbscheme` file in that directory and create an `upgrade.properties` file (as described above).
118+
119+
#### Doing the downgrade manually
120+
121+
The process is similar for downgrade scripts, but there is a reversal in terminology: your **new** dbscheme will now be the one called `old.dbscheme`.
122+
123+
1. Get a hash of your new `.dbscheme` file, with `git hash-object ql/lib/<mylang>.dbscheme`
124+
125+
2. Create a downgrade directory with that hash as its name, for example:
126+
```
127+
mkdir downgrades/9fdd1d40fd3c3f8f9db8fabf5a353580d14c663a
128+
```
129+
130+
3. Copy your new `.dbscheme` file to that directory, using the name `old.dbscheme`.
131+
```
132+
cp ql/lib/<mylang>.dbscheme ql/lib/upgrades/454f1e15151422355049dc4f1f0486a03baeffef/old.dbscheme
133+
```
134+
135+
4. Put a copy of the `.dbscheme` from `main` in that directory and create an `upgrade.properties` file that performs the downgrade (as described above).

ruby/doc/prepare-db-upgrade.md

Lines changed: 0 additions & 91 deletions
This file was deleted.

0 commit comments

Comments
 (0)