Skip to content

Commit aee6f11

Browse files
auxesisauxesis
committed
docs(dev): move all development docs into the development guide
1 parent 3cb9960 commit aee6f11

File tree

2 files changed

+235
-223
lines changed

2 files changed

+235
-223
lines changed

‎DEVELOPMENT.md

Lines changed: 234 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,239 @@
1+
# Developing CipherStash EQL
12

3+
## Table of Contents
24

5+
- [How this project is organised](#how-this-project-is-organised)
6+
- [Set up a local development environment](#set-up-a-local-development-environment)
7+
- [Installing mise](#installing-mise)
8+
- [Testing](#testing)
9+
- [Running tests locally](#running-tests-locally)
10+
- [Releasing](#releasing)
11+
12+
### How this project is organised
13+
14+
Development is managed through [mise](https://mise.jdx.dev/), both locally and [in CI](https://github.com/cipherstash/encrypt-query-language/actions).
15+
16+
mise has tasks for:
17+
18+
- Building EQL install and uninstall scripts (`build`)
19+
- Starting and stopping PostgreSQL containers (`postgres:up`, `postgres:down`)
20+
- Running unit and integration tests (`test`, `reset`)
21+
22+
These are the important files in the repo:
23+
24+
```
25+
.
26+
├── mise.toml <-- the main config file for mise
27+
├── tasks/ <-- mise tasks
28+
├── sql/ <-- The individual SQL components that make up EQL
29+
├── docs/ <-- Tutorial, reference, and concept documentation
30+
├── tests/ <-- Unit and integration tests
31+
│ ├── docker-compose.yml <-- Docker configuration for running PostgreSQL instances
32+
│ └── *.sql <-- Individual unit and integration tests
33+
├── release/ <-- Build artifacts produced by the `build` task
34+
├── examples/ <-- Example uses of EQL in different languages
35+
└── playground/ <-- Playground enviroment for experimenting with EQL and CipherStash Proxy
36+
```
37+
38+
## Set up a local development environment
39+
40+
> [!IMPORTANT]
41+
> **Before you follow this how-to** you need to have this software installed:
42+
> - [mise](https://mise.jdx.dev/) — see the [installing mise](#installing-mise) instructions
43+
> - [Docker](https://www.docker.com/) — see Docker's [documentation for installing](https://docs.docker.com/get-started/get-docker/)
44+
45+
Local development quickstart:
46+
47+
``` shell
48+
# Clone the repo
49+
git clone https://github.com/cipherstash/encrypt-query-language
50+
cd encrypt-query-language
51+
52+
# Install dependencies
53+
mise trust --yes
54+
55+
# Build EQL installer and uninstaller, outputting to release/
56+
mise run build
57+
58+
# Start a postgres instance (defaults to PostgreSQL 17)
59+
mise run postgres:up --extra-args "--detach --wait"
60+
61+
# Run the tests (defaults to PostgreSQL 17)
62+
mise run test
63+
64+
# Stop and remove all containers and networks
65+
mise run postgres:down
66+
```
67+
68+
### Installing mise
69+
70+
> [!IMPORTANT]
71+
> You must complete this step to set up a local development environment.
72+
73+
Local development and task running in CI is managed through [mise](https://mise.jdx.dev/).
74+
75+
To install mise:
76+
77+
- If you're on macOS, run `brew install mise`
78+
- If you're on another platform, check out the mise [installation methods documentation](https://mise.jdx.dev/installing-mise.html#installation-methods)
79+
80+
Then add mise to your shell:
81+
82+
```shell
83+
# If you're running Bash
84+
echo 'eval "$(mise activate bash)"' >> ~/.bashrc
85+
86+
# If you're running Zsh
87+
echo 'eval "$(mise activate zsh)"' >> ~/.zshrc
88+
```
89+
90+
We use [`cargo-binstall`](https://github.com/cargo-bins/cargo-binstall) for faster installation of tools installed via `mise` and Cargo.
91+
We install `cargo-binstall` via `mise` when installing development and testing dependencies.
92+
93+
> [!TIP]
94+
> We provide abbreviations for most of the commands that follow.
95+
> For example, `mise run postgres:setup` can be abbreviated to `mise r s`.
96+
> Run `mise tasks --extended` to see the task shortcuts.
97+
98+
## Testing
99+
100+
There are tests for checking EQL against PostgreSQL versions 14–17, that verify:
101+
102+
- Adding, removing, and modifying encrypted data and indexes
103+
- Validating, applying, and removing configuration for encrypted data and encrypted indexes
104+
- Validating schemas for EQL configuration, encrypted data, and encrypted indexes
105+
- Using PostgreSQL operators on encrypted data and indexes (`=`, `<>`, `@>`)
106+
107+
The easiest way to run the tests [is in GitHub Actions](./.github/workflows/test-eql.yml):
108+
109+
- Automatically whenever there are changes in the `sql/`, `tests/`, or `tasks/` directories
110+
- By manually running [the workflow](https://github.com/cipherstash/encrypt-query-language/actions/workflows/test-eql.yml)
111+
112+
This is how the `test-eql.yml` workflow functions:
113+
114+
```mermaid
115+
---
116+
title: Testing EQL
117+
---
118+
stateDiagram-v2
119+
direction LR
120+
classDef code font-family:monospace;
121+
122+
123+
state "🧍 Human makes changes to EQL sources" as changes
124+
state sources_fork <<fork>>
125+
state sources_join <<join>>
126+
state "sql/*.sql" as source_sql
127+
state "tasks/**/*" as source_tasks
128+
state "tests/**/*" as source_tests
129+
state sources_changed <<choice>>
130+
131+
state "🛠️ Trigger GitHub Actions workflow test-eql.yml" as build_triggered
132+
state "Matrix: Test EQL SQL components" as matrix
133+
state "Test with Postgres 14" as pg14
134+
state "Test with Postgres 15" as pg15
135+
state "Test with Postgres 16" as pg16
136+
state "Test with Postgres 17" as pg17
137+
state "Check build results" as check
138+
state if_state <<choice>>
139+
140+
changes --> sources_fork
141+
sources_fork --> source_sql:::code
142+
sources_fork --> source_tests:::code
143+
sources_fork --> source_tasks:::code
144+
source_sql --> sources_join
145+
source_tests --> sources_join
146+
source_tasks --> sources_join
147+
sources_join --> source_changed_check
148+
source_changed_check --> sources_changed
149+
sources_changed --> build_triggered : Some changes
150+
sources_changed --> [*]: No changes
151+
152+
state "Check source changes" as source_changed_check
153+
154+
[*] --> changes
155+
156+
build_triggered --> matrix
157+
158+
state fork_state <<fork>>
159+
matrix --> fork_state
160+
fork_state --> pg14
161+
fork_state --> pg15
162+
fork_state --> pg16
163+
fork_state --> pg17
164+
165+
state join_state <<join>>
166+
pg14 --> join_state
167+
pg15 --> join_state
168+
pg16 --> join_state
169+
pg17 --> join_state
170+
171+
state "✅ Pass build" as build_pass
172+
state "❌ Fail build" as build_fail
173+
join_state --> check
174+
check --> if_state
175+
if_state --> build_pass: All success
176+
if_state --> build_fail : Any failures
177+
build_pass --> [*]
178+
build_fail --> [*]
179+
```
180+
181+
You can also [run the tests locally](#running-tests-locally) when doing local development.
182+
183+
### Running tests locally
184+
185+
> [!IMPORTANT]
186+
> **Before you run the tests locally** you need to [set up a local dev environment](#set-up-a-local-development-environment).
187+
188+
To run tests locally with PostgreSQL 17:
189+
190+
``` shell
191+
# Start a postgres instance (defaults to PostgreSQL 17)
192+
mise run postgres:up --extra-args "--detach --wait"
193+
194+
# Run the tests (defaults to PostgreSQL 17)
195+
mise run test
196+
197+
# Stop and remove all containers and networks
198+
mise run postgres:down
199+
```
200+
201+
You can run the same tasks for Postgres 14, 15, 16, and 17 by specifying arguments:
202+
203+
```shell
204+
# Start a postgres 14 instance
205+
mise run postgres:up postgres-14 --extra-args "--detach --wait"
206+
207+
# Run the tests against postgres 14
208+
mise run test --postgres 14
209+
210+
# Stop postgres and remove all containers and networks
211+
mise run postgres:down
212+
```
213+
214+
The configuration for the Postgres containers in `tests/docker-compose.yml`.
215+
216+
Limitations:
217+
218+
- **Volumes for Postgres containers are not persistent.**
219+
If you need to look at data in the container, uncomment a volume in
220+
`tests/docker-compose.yml`
221+
- **You can't run multiple Postgres containers at the same time.**
222+
All the containers bind to the same port (`7543`). If you want to run
223+
multiple containers at the same time, you have to change the ports by
224+
editing `tests/docker-compose.yml`
225+
226+
## Releasing
227+
228+
To cut a [release](https://github.com/cipherstash/encrypt-query-language/releases) of EQL:
229+
230+
1. Draft a [new release](https://github.com/cipherstash/encrypt-query-language/releases/new) on GitHub.
231+
1. Choose a tag, and create a new one with the prefix `eql-` followed by a [semver](https://semver.org/) (for example, `eql-1.2.3`).
232+
1. Generate the release notes.
233+
1. Optionally set the release to be the latest (you can set a release to be latest later on if you are testing out a release first).
234+
1. Click `Publish release`.
235+
236+
This will trigger the [Release EQL](https://github.com/cipherstash/encrypt-query-language/actions/workflows/release-eql.yml) workflow, which will build and attach artifacts to [the release](https://github.com/cipherstash/encrypt-query-language/releases/).
3237

4238

5239
====

0 commit comments

Comments
 (0)