Merge pull request #422 from cagov/365_update_dbt_docs

ian-r-rose · web-flow · commit ebd59ff8c555 · 2024-11-15T12:09:18.000-08:00
added Snowflake OAuth instructions and fixed many case, spelling, and grammars errors
diff --git a/docs/code/local-setup.md b/docs/code/local-setup.md
@@ -48,7 +48,7 @@ We use [Terraform](https://www.terraform.io/) to manage infrastructure.
 Dependencies for Terraform (mostly in the [go ecosystem](https://go.dev/))
 can be installed via a number of different package managers.
 
-If you are running Mac OS, you can install you can install these dependencies with [Homebrew](https://brew.sh/).
+If you are running Mac OS, you can install these dependencies with [Homebrew](https://brew.sh/).
 First, install Homebrew
 
 ```bash
@@ -89,7 +89,7 @@ export SNOWFLAKE_WAREHOUSE=TRANSFORMING_XS_DEV
 export SNOWFLAKE_AUTHENTICATOR=ExternalBrowser
 ```
 
-This will enable you to perform transforming activities which is needed for dbt.
+This will enable you to perform transforming activities which are needed for dbt.
 Open a new terminal and verify that the environment variables are set.
 
 **Switch to Loader role**
@@ -104,7 +104,7 @@ export SNOWFLAKE_WAREHOUSE=LOADING_XS_DEV
 export SNOWFLAKE_AUTHENTICATOR=ExternalBrowser
 ```
 
-This will enable you to perform loading activities and is needed to which is needed for Airflow or Fivetran.
+This will enable you to perform loading activities which are needed for Airflow or Fivetran.
 Again, open a new terminal and verify that the environment variables are set.
 
 ## Configure AWS (optional)
diff --git a/docs/dbt/dbt-performance.md b/docs/dbt/dbt-performance.md
diff --git a/docs/infra/architecture.md b/docs/infra/architecture.md
@@ -131,15 +131,15 @@ There are six primary functional roles:
 
 ## Reporting and analysis
 
-The most prominent consumer of the data products from this project are PowerBI and Tableau dashboards and the CalInnovate team.
+The most prominent consumers of the data products from this project are PowerBI and Tableau dashboards and the CalInnovate team.
 
 ## Custom schema names
 
 dbt's default method for generating [custom schema names](https://docs.getdbt.com/docs/build/custom-schemas)
 works well for a single-database setup:
 
 * It allows development work to occur in a separate schema from production models.
-* It allows analytics engineers to develop side-by-side without stepping on each others toes.
+* It allows analytics engineers to develop side-by-side without stepping on each other's toes.
 
 A downside of the default is that production models all get a prefix,
 which may not be an ideal naming convention for end-users.
@@ -169,7 +169,7 @@ So your data access looks like the following:
 
 ![developer](../images/developer.png)
 
-Now let's consider the nigthly production build. This service account builds the production models
+Now let's consider the nightly production build. This service account builds the production models
 in `TRANSFORM_PRD` and `ANALYTICS_PRD` based on the raw data in `RAW_PRD`.
 The development environment effectively doesn't exist to this account, and data access looks like the following:
 
diff --git a/docs/infra/cloud-infrastructure.md b/docs/infra/cloud-infrastructure.md
@@ -1,4 +1,4 @@
-# Cloud Infrastructure
+# Cloud infrastructure
 
 The DSE team [uses Terraform](../code/terraform-local-setup.md) to manage cloud infrastructure.
 Our stack includes:
diff --git a/docs/infra/snowflake.md b/docs/infra/snowflake.md
@@ -82,15 +82,15 @@ X-Large: Powerful for demanding workloads and data-intensive operations.
 3X-Large: Triple the capacity of X-Large.
 4X-Large: Quadruple the capacity of X-Large.
 
-1. **`LOADING_{size}_{env}`**: These warehouse is for loading data to `RAW`.
+1. **`LOADING_{size}_{env}`**: This warehouse is for loading data to `RAW`.
 1. **`TRANSFORMING_{size}_{env}`**: This warehouse is for transforming data in `TRANSFORM` and `ANALYTICS`.
 1. **`REPORTING_{size}_{env}`**: This warehouse is the role for BI tools and other end-users of the data.
 
 ### Four roles
 
 There are four primary functional roles:
 
-1. **`LOADER_{env}`**: This role is for tooling like Fivetran or Airflow to load raw data in to the `RAW` database.
+1. **`LOADER_{env}`**: This role is for tooling like Fivetran or Airflow to load raw data into the `RAW` database.
 1. **`TRANSFORMER_{env}`**: This is the analytics engineer/dbt role, for transforming raw data into something analysis-ready. It has read/write/control access to both `TRANSFORM` and `ANALYTICS`, and read access to `RAW`.
 1. **`REPORTER_{env}`**: This role read access to `ANALYTICS`, and is intended for BI tools and other end-users of the data.
 1. **`READER_{env}`**: This role has read access to all three databases, and is intended for CI service accounts to generate documentation.
diff --git a/docs/learning/cloud-data-warehouses.md b/docs/learning/cloud-data-warehouses.md
@@ -1,4 +1,4 @@
-# Cloud Data Warehouses
+# Cloud data warehouses
 
 ## What is a cloud data warehouse?
 
diff --git a/docs/learning/git.md b/docs/learning/git.md
@@ -1,4 +1,4 @@
-# Git and GitHub
+# git and GitHub
 
 ## What are git and GitHub?
 
@@ -42,7 +42,7 @@ for learning to use its various features, including:
 * [How to review pull requests](https://github.com/skills/review-pull-requests)
 * [How to automate tasks and use CI/CD with GitHub actions](https://github.com/skills/hello-github-actions)
 
-## Git+GitHub at CalData
+## git and GitHub at CalData
 
 On the CalData Data Services and Engineering team we make heavy use of git and GitHub for our projects,
 and have our own set of [guidelines and best practices](../code/code-review.md) for code review.
diff --git a/docs/learning/glossary.md b/docs/learning/glossary.md
@@ -1,4 +1,4 @@
-# Modern Data Stack Glossary
+# Modern data stack glossary
 
 This glossary is a reference for commonly used acronyms, terms, and tools associated with the modern data stack and data and analytics engineering practices.
 
diff --git a/docs/learning/naming-conventions.md b/docs/learning/naming-conventions.md
@@ -1,8 +1,8 @@
-# Naming Conventions
+# Naming conventions
 
 This page documents the Data Services and Engineering (DSE) team's naming conventions for cloud resources.
 
-## General Approach
+## General approach
 
 Our approach is adapted from [this blog post](https://stepan.wtf/cloud-naming-convention/).
 The goals of establishing a naming convention are:
diff --git a/docs/learning/security.md b/docs/learning/security.md
@@ -1,9 +1,9 @@
-# Security Guidelines
+# Security guidelines
 
 This document describes security conventions for CalData's Data Services and Engineering team,
 especially as it relates to cloud and SaaS services.
 
-## Cloud Security and IAM
+## Cloud security and IAM
 
 The major public clouds (AWS, GCP, Azure) all have a service for Identity and Access Management (IAM).
 This allows us to manage which users or services are able to perform actions on which resources.
@@ -17,7 +17,7 @@ In general, IAM is described by:
 
 Most of the work of IAM is managing users, permissions, groups, policies, and roles to perform tasks in a secure way.
 
-### Principle of Least-Privilege
+### Principle of least-privilege
 
 In general, users and roles should be assigned permissions according to the
 [Principle of Least Privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege),
@@ -48,11 +48,11 @@ Some good practices around the use of service accounts
   and being able to edit or decommission accounts separately from each other is a good idea.
 * Regularly rotate access keys for long-term service accounts.
 
-### Production and Development Environments
+### Production and development environments
 
 Production environments should be treated with greater care than development ones.
 In the testing and developing of a service, roles and policies are often crafted
-which do not follow the principal of least privilege (i.e., they have too many permissions).
+which do not follow the principle of least privilege (i.e., they have too many permissions).
 
 When productionizing a service or application, make sure to review the relevant
 roles and service accounts to ensure they only have the necessary policies,
@@ -72,7 +72,7 @@ GCS buckets in a project, when their application only requires access to one.
 AWS has a nice [user guide](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html)
 for how to work with IAM, including some [best-practices](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html).
 
-## Third-party SaaS Integrations
+## Third-party SaaS integrations
 
 Often a third-party software-as-a-service (SaaS) provider will require service accounts
 to access resources within a cloud account.
@@ -116,6 +116,6 @@ The DSE team both manages CalData projects and onboards clients into Fivetran, a
 Security policies for Snowflake can be found
 [here](../infra/snowflake.md#security-policies).
 
-## Security Review Standard Practices
+## Security review standard practices
 
 (TODO, possibly pulling from [Agile Application Security](https://www.amazon.com/Agile-Application-Security-Enabling-Continuous/dp/1491938846/ref=cm_cr_arp_d_product_top))
diff --git a/docs/setup/dbt-setup.md b/docs/setup/dbt-setup.md
@@ -1,19 +1,24 @@
-# dbt Project Setup
+# dbt project setup
 
 To set up a new project on dbt Cloud follow these steps:
 
 1. Give your new project a name.
 1. Click *Advanced settings* and in the *Project subdirectory* field, enter "transform"
 1. Select a data warehouse connection. (e.g. Snowflake, BigQuery, Redshift)
-1. For the *Development credentials* section you'll want to:
-    1. Under *Auth method* select *Key pair*
-    1. Enter your data warehouse username
-    1. Enter the private key amd private key passphrase
-    1. For more guidance, read [dbt's docs on connecting to Snowflake via key pair](https://docs.getdbt.com/docs/cloud/connect-data-platform/connect-snowflake#key-pair)
+1. For the *Development credentials* section you'll want to choose between Snowflake OAuth or Key pair. In general, Snowflake OAuth is preferred for human users (which is what the development environment is for). It is also an enterprise dbt Cloud feature, so if working with a standard account, you'll need to use key pair.
+
+    1. For Snowflake OAuth:
+        1. Follow dbt's instructions for set up [here](https://docs.getdbt.com/docs/cloud/manage-access/set-up-snowflake-oauth)
+
+    1. For Key pair:
+        1. Under *Auth method* select *Key pair*
+        1. Enter your data warehouse username
+        1. Enter the private key and private key passphrase
+        1. For more guidance, read [dbt's docs on connecting to Snowflake via key pair](https://docs.getdbt.com/docs/cloud/connect-data-platform/connect-snowflake#key-pair)
 1. Finally click the *Test Connection* button.
 1. Connect the appropriate repository (usually GitHub). Read [dbt's docs on connecting to GitHub](https://docs.getdbt.com/docs/cloud/git/connect-github).
 
-Once you're through the first five steps you can return to the dbt homepage and click the Settings button in the upper right corner. From there you can follow the steps to configure three environments for Continuous intergation - CI, development, and production. Read [dbt's docs on CI in dbt Cloud](https://docs.getdbt.com/docs/deploy/continuous-integration). Read [dbt's docs on creating production (deployment) environments](https://docs.getdbt.com/docs/deploy/deploy-environments) and [dbt's docs on creating and scheduling deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs#create-and-schedule-jobs).
+Once you're through the first five steps you can return to the dbt homepage and click the Settings button in the upper right corner. From there you can follow the steps to configure three environments for Continuous integration - CI, development, and production. Read [dbt's docs on CI in dbt Cloud](https://docs.getdbt.com/docs/deploy/continuous-integration). Read [dbt's docs on creating production (deployment) environments](https://docs.getdbt.com/docs/deploy/deploy-environments) and [dbt's docs on creating and scheduling deploy jobs](https://docs.getdbt.com/docs/deploy/deploy-jobs#create-and-schedule-jobs).
 
 You'll also want to [configure notifications for job failures](https://docs.getdbt.com/docs/deploy/job-notifications).
 
diff --git a/docs/setup/snowflake-setup.md b/docs/setup/snowflake-setup.md
@@ -1,4 +1,4 @@
-# New Project Setup
+# New project setup
 
 The DSE team regularly creates new Snowflake accounts in our Snowflake org.
 We do this instead of putting all of our data projects into our main account for a few reasons:
@@ -64,7 +64,7 @@ conda install -c conda-forge terraform tflint  # Install terraform and tflint
     1. Select "Business Critical" for the Snowflake Edition.
     1. You will be prompted to create an initial user with `ACCOUNTADMIN` privileges. This should be you.
         You will be prompted to create a password for your user. Create one using your password manager,
-        but know that it will ask you change your password upon first log-in.
+        but know that it will ask you to change your password upon first log-in.
     1. Save the Account Locator and Account URL for your new account.
 1. Log into your new account. You should be prompted to change your password. Save the updated password in your password manager.
 
diff --git a/mkdocs.yml b/mkdocs.yml
@@ -38,34 +38,34 @@ markdown_extensions:
 
 nav:
   - Introduction: index.md
-  - Code Development:
-      - Local Environment Setup: code/local-setup.md
+  - Code development:
+      - Local environment setup: code/local-setup.md
       - Codespaces: code/codespaces.md
-      - Code Review: code/code-review.md
-      - Writing Documentation: code/writing-documentation.md
-      - Terraform Setup: code/terraform-local-setup.md
-  - Project Infrastructure:
-      - Cloud Infrastructure: infra/cloud-infrastructure.md
-      - Project Architecture: infra/architecture.md
-      - Snowflake Overview: infra/snowflake.md
-  - Project Setup:
-      - Snowflake Setup: setup/snowflake-setup.md
-      - Git/Github Setup: setup/repo-setup.md
-      - Terraform Setup: setup/terraform-project-setup.md
-      - Sentinel Setup: setup/sentinel-setup.md
-      - dbt Setup: setup/dbt-setup.md
-      - Adding Service Accounts: setup/snowflake-service-accounts.md
-      - Project Teardown: setup/project-teardown.md
+      - Code review: code/code-review.md
+      - Writing documentation: code/writing-documentation.md
+      - Terraform setup: code/terraform-local-setup.md
+  - Project infrastructure:
+      - Cloud infrastructure: infra/cloud-infrastructure.md
+      - Project architecture: infra/architecture.md
+      - Snowflake overview: infra/snowflake.md
+  - Project setup:
+      - Snowflake setup: setup/snowflake-setup.md
+      - git/Github setup: setup/repo-setup.md
+      - Terraform setup: setup/terraform-project-setup.md
+      - Sentinel setup: setup/sentinel-setup.md
+      - dbt Cloud setup: setup/dbt-setup.md
+      - Adding service accounts: setup/snowflake-service-accounts.md
+      - Project teardown: setup/project-teardown.md
   - dbt:
-      - dbt Overview: dbt/dbt.md
-      - dbt Performance: dbt/dbt-performance.md
-      - dbt Snowflake Project: dbt_docs_snowflake/index.html
+      - dbt overview: dbt/dbt.md
+      - dbt performance: dbt/dbt-performance.md
+      - dbt Cloud Snowflake project: dbt_docs_snowflake/index.html
   - Data:
-      - Building Footprints: data/footprints.md
+      - Building footprints: data/footprints.md
   - Learning:
-      - MDSA Glossary: learning/glossary.md
-      - Security Guidelines: learning/security.md
-      - Naming Conventions: learning/naming-conventions.md
+      - MDSA glossary: learning/glossary.md
+      - Security guidelines: learning/security.md
+      - Naming conventions: learning/naming-conventions.md
       - git: learning/git.md
       - dbt: learning/dbt.md
-      - Cloud Data Warehouses: learning/cloud-data-warehouses.md
+      - Cloud data warehouses: learning/cloud-data-warehouses.md

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-# Cloud Infrastructure`
	`1`	`+# Cloud infrastructure`
`2`	`2`
`3`	`3`	`The DSE team [uses Terraform](../code/terraform-local-setup.md) to manage cloud infrastructure.`
`4`	`4`	`Our stack includes:`
Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-# Cloud Data Warehouses`
	`1`	`+# Cloud data warehouses`
`2`	`2`
`3`	`3`	`## What is a cloud data warehouse?`
`4`	`4`
Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,4 @@`
`1`		`-# Modern Data Stack Glossary`
	`1`	`+# Modern data stack glossary`
`2`	`2`
`3`	`3`	`This glossary is a reference for commonly used acronyms, terms, and tools associated with the modern data stack and data and analytics engineering practices.`
`4`	`4`