Skip to content

Add: Dashboard db migrations & tooling #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Mar 13, 2025
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,6 @@ NEXT_PUBLIC_DEFAULT_API_DOMAIN=e2b.dev
NEXT_PUBLIC_EXPOSE_STORYBOOK=0
NEXT_PUBLIC_SCAN=0
NEXT_PUBLIC_MOCK_DATA=0

# For applying migrations
# POSTGRES_CONNECTION_STRING=
104 changes: 100 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,21 +75,28 @@ vercel storage add
3. Copy the `anon key` and `service_role key`
4. Copy the project URL

#### c. Supabase Storage Setup
#### c. Database Setup
1. Retrieve the `POSTGRES_CONNECTION_STRING` from the Supabase project settings
2. Run the migrations by running the following command:
```bash
bun run db:migrations:apply
```

#### d. Supabase Storage Setup
1. Go to Storage > Buckets
2. Create a new **public** bucket named `profile-pictures`
3. Apply storage access policies by running the SQL from [supabase/policies/buckets.sql](supabase/policies/buckets.sql) in the Supabase SQL Editor:
3. Apply storage access policies by running the SQL from [migrations/supabase/profile-picture-bucket.sql](migrations/supabase/profile-picture-bucket.sql) in the Supabase SQL Editor:
- These policies ensure only Supabase admin (service role) can write to and list files in the bucket
- Public URLs are accessible for downloading files if the exact path is known
- Regular users cannot browse, upload, update, or delete files in the bucket

#### d. Environment Variables
#### e. Environment Variables
```bash
# Copy the example env file
cp .env.example .env.local
```

#### e. Cookie Encryption
#### f. Cookie Encryption
The dashboard uses encrypted cookies for secure data storage. You'll need to set up a `COOKIE_ENCRYPTION_KEY`:

```bash
Expand Down Expand Up @@ -179,3 +186,92 @@ If you need help or have questions:
This project is licensed under the Apache License, Version 2.0 - see the [LICENSE](LICENSE) file for details.

Copyright 2025 FoundryLabs, Inc.

## Database Migrations

The dashboard uses a custom migration system to manage database schema changes in a controlled, versioned manner.

### Migration System Overview

- **Timestamp-based**: Migrations are ordered by timestamp prefixes (YYYYMMDDHHMMSS)
- **Idempotent**: Each migration is applied only once
- **Transactional**: Migrations run in transactions for atomic changes
- **Integrity checks**: Checksums verify migrations haven't been modified after application
- **SQL-native**: Write pure SQL migrations with full PostgreSQL feature support

### Creating Migrations

To create a new migration:

```bash
# Create a migration with a description
bun run db:migrations:create "add user profiles"
# Creates: migrations/20250315123045.sql with description comment

# Or create without description
bun run db:migrations:create
# Creates: migrations/20250315123045.sql
```

### Writing Migrations

Migrations are SQL files that can contain multiple statements, including complex PostgreSQL features:

```sql
-- Migration: add user profiles
-- Timestamp: 20250315123045

-- Create profiles table
CREATE TABLE profiles (
id UUID PRIMARY KEY REFERENCES auth.users(id),
display_name TEXT,
bio TEXT,
created_at TIMESTAMP WITH TIME ZONE DEFAULT CURRENT_TIMESTAMP
);

-- Create function to handle profile creation
CREATE OR REPLACE FUNCTION create_profile_for_new_user()
RETURNS TRIGGER AS $$
BEGIN
INSERT INTO profiles (id)
VALUES (NEW.id);
RETURN NEW;
END;
$$ LANGUAGE plpgsql;

-- Set up trigger
CREATE TRIGGER on_user_created
AFTER INSERT ON auth.users
FOR EACH ROW
EXECUTE FUNCTION create_profile_for_new_user();
```

### Applying Migrations

To apply pending migrations:

```bash
bun run db:migrations:apply
```

This will:
1. Create the migrations tracking table if it doesn't exist
2. Apply any migrations that haven't been run yet
3. Skip already-applied migrations
4. Provide a summary of applied/skipped migrations

### Migration Safety

The system includes several safety features:
- Migrations are applied in a transaction (all-or-nothing)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not true!

- Each migration is recorded with a checksum to detect modifications
- Warnings are shown if a previously applied migration file has changed
- Execution stops on the first error to prevent partial migrations

### Best Practices

- **One change per migration**: Keep migrations focused on a single logical change
- **Backward compatibility**: When possible, design migrations to be backward compatible
- **Comments**: Document the purpose of complex migrations
- **Testing**: Test migrations in development before applying to production
- **Avoid modifying applied migrations**: Create a new migration instead of changing an existing one
12 changes: 7 additions & 5 deletions bun.lock
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
"@tanstack/match-sorter-utils": "^8.19.4",
"@tanstack/react-query": "^5.65.0",
"@tanstack/react-table": "^8.20.6",
"@tanstack/react-virtual": "^3.13.2",
"@tanstack/react-virtual": "^3.11.3",
"@theguild/remark-mermaid": "^0.2.0",
"@types/mdx": "^2.0.13",
"@vercel/kv": "^3.0.0",
Expand All @@ -53,7 +53,7 @@
"next": "^15.2.2-canary.6",
"next-logger": "^5.0.1",
"next-themes": "^0.4.4",
"pg": "^8.13.1",
"pg": "^8.14.0",
"pino": "^9.6.0",
"pino-pretty": "^13.0.0",
"postgres": "^3.4.5",
Expand Down Expand Up @@ -93,7 +93,7 @@
"@tailwindcss/postcss": "^4.0.6",
"@testing-library/jest-dom": "^6.6.3",
"@testing-library/react": "^16.2.0",
"@types/bun": "latest",
"@types/bun": "^1.2.5",
"@types/node": "22.10.10",
"@types/pg": "^8.11.11",
"@types/react": "^19.0.8",
Expand Down Expand Up @@ -2574,7 +2574,7 @@

"pend": ["pend@1.2.0", "", {}, "sha512-F3asv42UuXchdzt+xXqfW1OGlVBe+mxa2mqI0pg5yAHZPvFmY3Y6drSf/GQ1A86WgWEN9Kzh/WrgKa6iGcHXLg=="],

"pg": ["pg@8.13.3", "", { "dependencies": { "pg-connection-string": "^2.7.0", "pg-pool": "^3.7.1", "pg-protocol": "^1.7.1", "pg-types": "^2.1.0", "pgpass": "1.x" }, "optionalDependencies": { "pg-cloudflare": "^1.1.1" }, "peerDependencies": { "pg-native": ">=3.0.1" }, "optionalPeers": ["pg-native"] }, "sha512-P6tPt9jXbL9HVu/SSRERNYaYG++MjnscnegFh9pPHihfoBSujsrka0hyuymMzeJKFWrcG8wvCKy8rCe8e5nDUQ=="],
"pg": ["pg@8.14.0", "", { "dependencies": { "pg-connection-string": "^2.7.0", "pg-pool": "^3.8.0", "pg-protocol": "^1.8.0", "pg-types": "^2.1.0", "pgpass": "1.x" }, "optionalDependencies": { "pg-cloudflare": "^1.1.1" }, "peerDependencies": { "pg-native": ">=3.0.1" }, "optionalPeers": ["pg-native"] }, "sha512-nXbVpyoaXVmdqlKEzToFf37qzyeeh7mbiXsnoWvstSqohj88yaa/I/Rq/HEVn2QPSZEuLIJa/jSpRDyzjEx4FQ=="],

"pg-cloudflare": ["pg-cloudflare@1.1.1", "", {}, "sha512-xWPagP/4B6BgFO+EKz3JONXv3YDgvkbVrGw2mTo3D6tVDQRh1e7cqVGvyR3BE+eQgAvx1XhW/iEASj4/jCWl3Q=="],

Expand All @@ -2584,7 +2584,7 @@

"pg-numeric": ["pg-numeric@1.0.2", "", {}, "sha512-BM/Thnrw5jm2kKLE5uJkXqqExRUY/toLHda65XgFTBTFYZyopbKjBe29Ii3RbkvlsMoFwD+tHeGaCjjv0gHlyw=="],

"pg-pool": ["pg-pool@3.7.1", "", { "peerDependencies": { "pg": ">=8.0" } }, "sha512-xIOsFoh7Vdhojas6q3596mXFsR8nwBQBXX5JiV7p9buEVAGqYL4yFzclON5P9vFrpu1u7Zwl2oriyDa89n0wbw=="],
"pg-pool": ["pg-pool@3.8.0", "", { "peerDependencies": { "pg": ">=8.0" } }, "sha512-VBw3jiVm6ZOdLBTIcXLNdSotb6Iy3uOCwDGFAksZCXmi10nyRvnP2v3jl4d+IsLYRyXf6o9hIm/ZtUzlByNUdw=="],

"pg-protocol": ["pg-protocol@1.7.1", "", {}, "sha512-gjTHWGYWsEgy9MsY0Gp6ZJxV24IjDqdpTW7Eh0x+WfJLFsm/TJx1MzL6T0D88mBvkpxotCQ6TwW6N+Kko7lhgQ=="],

Expand Down Expand Up @@ -3500,6 +3500,8 @@

"path-scurry/lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="],

"pg/pg-protocol": ["pg-protocol@1.8.0", "", {}, "sha512-jvuYlEkL03NRvOoyoRktBK7+qU5kOvlAwvmrH8sr3wbLrOdVWsRxQfz8mMy9sZFsqJ1hEWNfdWKI4SAmoL+j7g=="],

"pg/pg-types": ["pg-types@2.2.0", "", { "dependencies": { "pg-int8": "1.0.1", "postgres-array": "~2.0.0", "postgres-bytea": "~1.0.0", "postgres-date": "~1.0.4", "postgres-interval": "^1.1.0" } }, "sha512-qTAAlrEsl8s4OiEQY69wDvcMIdQN6wdz5ojQiOy6YRMuynxenON0O5oCpJI6lshc6scgAY8qvJ2On/p+CXY0GA=="],

"pkg-dir/find-up": ["find-up@6.3.0", "", { "dependencies": { "locate-path": "^7.1.0", "path-exists": "^5.0.0" } }, "sha512-v2ZsoEuVHYy8ZIlYqwPe/39Cy+cFDzp4dXPaxNvkEuouymu+2Jbz0PxpKarJHYJTmv2HWT3O382qY8l4jMWthw=="],
Expand Down
109 changes: 109 additions & 0 deletions migrations/20250205180205.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
/*
This migration adds team slugs and profile pictures to support user-friendly URLs and team branding.

It performs the following steps:

1. Adds two new columns to the teams table:
- slug: A URL-friendly version of the team name (e.g. "acme-inc")
- profile_picture_url: URL to the team's profile picture

2. Creates a slug generation function that:
- Takes a team name and converts it to a URL-friendly format
- Removes special characters, accents, and spaces
- Handles email addresses by only using the part before @
- Converts to lowercase and replaces spaces with hyphens

3. Installs the unaccent PostgreSQL extension for proper accent handling

4. Generates initial slugs for all existing teams:
- Uses the team name as base for the slug
- If multiple teams would have the same slug, appends part of the team ID
to ensure uniqueness

5. Sets up automatic slug generation for new teams:
- Creates a trigger that runs before team insertion
- Generates a unique slug using random suffixes if needed
- Only generates a slug if one isn't explicitly provided

6. Enforces slug uniqueness with a database constraint
*/

ALTER TABLE teams
ADD COLUMN slug TEXT,
ADD COLUMN profile_picture_url TEXT;

CREATE OR REPLACE FUNCTION generate_team_slug(name TEXT)
RETURNS TEXT AS $$
DECLARE
base_name TEXT;
BEGIN
base_name := SPLIT_PART(name, '@', 1);

RETURN LOWER(
REGEXP_REPLACE(
REGEXP_REPLACE(
UNACCENT(TRIM(base_name)),
'[^a-zA-Z0-9\s-]',
'',
'g'
),
'\s+',
'-',
'g'
)
);
END;
$$ LANGUAGE plpgsql;

CREATE EXTENSION IF NOT EXISTS unaccent;

WITH numbered_teams AS (
SELECT
id,
name,
generate_team_slug(name) as base_slug,
ROW_NUMBER() OVER (PARTITION BY generate_team_slug(name) ORDER BY created_at) as slug_count
FROM teams
WHERE slug IS NULL
)
UPDATE teams
SET slug =
CASE
WHEN t.slug_count = 1 THEN t.base_slug
ELSE t.base_slug || '-' || SUBSTRING(teams.id::text, 1, 4)
END
FROM numbered_teams t
WHERE teams.id = t.id;

CREATE OR REPLACE FUNCTION generate_team_slug_trigger()
RETURNS TRIGGER AS $$
DECLARE
base_slug TEXT;
test_slug TEXT;
suffix TEXT;
BEGIN
IF NEW.slug IS NULL THEN
base_slug := generate_team_slug(NEW.name);
test_slug := base_slug;

WHILE EXISTS (SELECT 1 FROM teams WHERE slug = test_slug) LOOP
suffix := SUBSTRING(gen_random_uuid()::text, 33, 4);
test_slug := base_slug || '-' || suffix;
END LOOP;

NEW.slug := test_slug;
END IF;
RETURN NEW;
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER team_slug_trigger
BEFORE INSERT ON teams
FOR EACH ROW
EXECUTE FUNCTION generate_team_slug_trigger();

ALTER TABLE teams
ADD CONSTRAINT teams_slug_unique UNIQUE (slug);

ALTER TABLE teams
ALTER COLUMN slug SET NOT NULL;
10 changes: 10 additions & 0 deletions migrations/20250311144556.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
CREATE OR REPLACE VIEW public.auth_users AS
SELECT
id,
email
FROM auth.users;

-- Revoke all permissions to ensure no public access
REVOKE ALL ON public.auth_users FROM PUBLIC;
REVOKE ALL ON public.auth_users FROM anon;
REVOKE ALL ON public.auth_users FROM authenticated;
14 changes: 6 additions & 8 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -10,27 +10,25 @@
"preview": "bun run build && bun start",
"lint": "next lint",
"lint:fix": "next lint --fix",

"<<<<<< Development Tools": "",
"dev:scan": "bun dev & bun scan",
"start:scan": "bun start & bun scan",

"<<<<<< Database": "",
"db:types": "bunx supabase@latest gen types typescript --schema public > src/types/database.types.ts --project-id $SUPABASE_PROJECT_ID",
"db:migration": "bun scripts/create-migration.ts",

"db:migrations:create": "bun run scripts:create-migration",
"db:migrations:apply": "bun run scripts:apply-migrations",
"<<<<<< Scripts": "",
"scripts:check-app-env": "bun scripts/check-app-env.ts",
"scripts:build-storybook": "bun scripts/build-storybook.ts",
"scripts:check-e2e-env": "bun scripts/check-e2e-env.ts",
"scripts:check-all-env": "bun scripts:check-app-env && bun scripts:check-e2e-env",

"scripts:create-migration": "bun scripts/create-migration.ts",
"scripts:apply-migrations": "bun scripts/apply-migrations.ts",
"<<<<<< Development": "",
"storybook": "storybook dev -p 6006",
"shad": "bunx shadcn@canary",
"prebuild": "bun scripts:build-storybook",
"postinstall": "fumadocs-mdx",

"<<<<<< Testing": "",
"test:run": "bun scripts:check-all-env && vitest run",
"test:integration": "bun scripts:check-app-env && vitest run src/__test__/integration/",
Expand Down Expand Up @@ -88,7 +86,7 @@
"next": "^15.2.2-canary.6",
"next-logger": "^5.0.1",
"next-themes": "^0.4.4",
"pg": "^8.13.1",
"pg": "^8.14.0",
"pino": "^9.6.0",
"pino-pretty": "^13.0.0",
"postgres": "^3.4.5",
Expand Down Expand Up @@ -128,7 +126,7 @@
"@tailwindcss/postcss": "^4.0.6",
"@testing-library/jest-dom": "^6.6.3",
"@testing-library/react": "^16.2.0",
"@types/bun": "latest",
"@types/bun": "^1.2.5",
"@types/node": "22.10.10",
"@types/pg": "^8.11.11",
"@types/react": "^19.0.8",
Expand Down
Loading