Skip to content

MVP ETL Module #90

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 8 commits into
base: main
Choose a base branch
from
Draft

MVP ETL Module #90

wants to merge 8 commits into from

Conversation

dehume
Copy link
Collaborator

@dehume dehume commented Apr 3, 2025 •
edited
Loading

Outline of code for ETL course MVP

Lesson 3 - Importing static files

  • Uses Duckdb
  • Dagster focus
    • Asset checks for ETL
    • Partitions
    • Tradeoffs with schedules vs sensors
    • How to use partitions with schedules and sensors

Lesson 4 - Importing data from APIs

  • Uses Duckdb
  • Dagster focus
    • Resources
    • Staging data before loading it

Lesson 5 - Importing static files/APIs with embedded ETL (dlt)

  • Uses dlt, Duckdb
  • Dagster focus
    • Embedded ETL and its benefits

Lesson 6 - Importing CDC (Sling)

  • Uses sling, Duckdb
  • Dagster focus
    • CDC data

Lesson 7 - Components

  • Uses sling, Duckdb
  • Dagster focus
    • Components

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 3, 2025 •
edited
Loading

Deploy preview for dagster-university ready!

✅ Preview
https://dagster-university-bjxa6g3us-elementl.vercel.app
https://dennis-ce-814-etl-code-example.dagster-university.dagster-docs.io

Built with commit 6d1b59d.
This pull request is being automatically deployed with vercel-action

github-advanced-security[bot]
github-advanced-security bot found potential problems May 12, 2025
View reviewed changes
.github/workflows/quality-check-dagster-and-etl.yml
Comment on lines +14 to +20
if: github.event.pull_request.draft == false
uses: ./.github/workflows/template-quality-check.yml
with:
working_directory: ./dagster_university/dagster_and_etl
# TODO: Disable integration tests from GHA
# postgres image has no windows/amd64
windows_pytest_cmd: uv run pytest dagster_and_etl/completed -v -m "not integration"

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI about 10 hours ago

To fix the issue, we will add a permissions block at the root level of the workflow. Since this workflow appears to perform quality checks and does not seem to require write access, we will set the permissions to contents: read. This ensures that the GITHUB_TOKEN has only the minimal permissions necessary for the workflow to execute.

Suggested changeset 1
.github/workflows/quality-check-dagster-and-etl.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/quality-check-dagster-and-etl.yml b/.github/workflows/quality-check-dagster-and-etl.yml
--- a/.github/workflows/quality-check-dagster-and-etl.yml
+++ b/.github/workflows/quality-check-dagster-and-etl.yml
@@ -2,2 +2,5 @@
 
+permissions:
+  contents: read
+
 on:
EOF
@@ -2,2 +2,5 @@

permissions:
contents: read

on:
Copilot is powered by AI and may make mistakes. Always verify output.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dehume