Skip to content

Commit 2a9467e

Browse files
fivetran-jamiefivetran-jamie
authored
Merge pull request #1 from fivetran/dev-jamie
staging package build
2 parents 4d18bac + 3757ebb commit 2a9467e

File tree

84 files changed

+2803
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

84 files changed

+2803
-0
lines changed

‎.circleci/config.yml‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
version: 2
2+
3+
jobs:
4+
build:
5+
docker:
6+
- image: circleci/python:3.7.9-stretch
7+
8+
steps:
9+
- checkout
10+
11+
- run:
12+
run: setup_creds
13+
command: |
14+
echo $GCLOUD_SERVICE_KEY | base64 --decode --ignore-garbage > ${HOME}/gcloud-service-key.json
15+
- run:
16+
name: "Setup dbt"
17+
command: |
18+
sudo apt install libsasl2-dev
19+
python3 -m venv venv
20+
. venv/bin/activate
21+
pip install --upgrade pip setuptools
22+
pip install -r integration_tests/requirements.txt
23+
mkdir -p ~/.dbt
24+
cp integration_tests/ci/sample.profiles.yml ~/.dbt/profiles.yml
25+
- run:
26+
name: "Run Tests - Spark"
27+
command: |
28+
. venv/bin/activate
29+
echo `pwd`
30+
cd integration_tests
31+
dbt deps
32+
dbt seed --target spark --full-refresh
33+
dbt run --target spark --full-refresh
34+
dbt run --target spark
35+
dbt test --target spark
36+
- run:
37+
name: "Run Tests - Snowflake"
38+
command: |
39+
. venv/bin/activate
40+
echo `pwd`
41+
cd integration_tests
42+
dbt deps
43+
dbt seed --target snowflake --full-refresh
44+
dbt run --target snowflake --full-refresh
45+
dbt test --target snowflake
46+
- run:
47+
name: "Run Tests - BigQuery"
48+
environment:
49+
GCLOUD_SERVICE_KEY_PATH: "/home/circleci/gcloud-service-key.json"
50+
51+
command: |
52+
. venv/bin/activate
53+
echo `pwd`
54+
cd integration_tests
55+
dbt deps
56+
dbt seed --target bigquery
57+
dbt run --target bigquery --full-refresh
58+
dbt test --target bigquery
59+
- run:
60+
name: "Run Tests - Postgres"
61+
command: |
62+
. venv/bin/activate
63+
echo `pwd`
64+
cd integration_tests
65+
dbt deps
66+
dbt seed --target postgres --full-refresh
67+
dbt run --target postgres --full-refresh
68+
dbt run --target postgres
69+
dbt test --target postgres
70+
- run:
71+
name: "Run Tests - Redshift"
72+
command: |
73+
. venv/bin/activate
74+
echo `pwd`
75+
cd integration_tests
76+
dbt deps
77+
dbt seed --target redshift --full-refresh
78+
dbt run --target redshift --full-refresh
79+
dbt test --target redshift
80+
- save_cache:
81+
key: deps2-{{ .Branch }}
82+
paths:
83+
- "venv"

‎.github/ISSUE_TEMPLATE/bug_report.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
---
2+
name: Bug report
3+
about: Create a report to help us improve
4+
title: BUG - [your bug title here]
5+
labels: bug
6+
assignees: ''
7+
8+
---
9+
10+
**Are you a current Fivetran customer?**
11+
<!--- Please tell us your name, title and company -->
12+
13+
14+
**Describe the bug**
15+
<!--- A clear and concise description of what the bug is. -->
16+
17+
**Steps to reproduce**
18+
1. Go to '...'
19+
2. Click on '....'
20+
3. Scroll down to '....'
21+
4. See error
22+
23+
**Expected behavior**
24+
<!--- A clear and concise description of what you expected to happen. -->
25+
26+
**Project variables configuration**
27+
<!--- Copy and paste the contents of your `dbt_project.yml` file below. -->
28+
```yml
29+
copy dbt_project.yml here
30+
```
31+
32+
**Package Version**
33+
<!---Copy and paste the contents of your `packages.yml` file below.-->
34+
```yml
35+
copy packages.yml here
36+
```
37+
38+
**Warehouse**
39+
<!--- Which warehouse you are running dbt and this package against. -->
40+
<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). -->
41+
- [ ] BigQuery
42+
- [ ] Redshift
43+
- [ ] Snowflake
44+
- [ ] Postgres
45+
- [ ] Databricks
46+
- [ ] Other (provide details below)
47+
48+
**Additional context**
49+
<!--- Add any other context about the problem here. -->
50+
51+
**Screenshots**
52+
<!--- If applicable, add screenshots to help explain your problem. -->
53+
54+
**Please indicate the level of urgency**
55+
<!--- Is this impacting your reporting of a critical dashboard? Are you just testing this for the first time? Provide us context so that we can prioritize accordingly. -->
56+
57+
**Are you interested in contributing to this package?**
58+
<!--- Our packages are all open-source and we welcome contributions from the community! Are you interested in helping us fix this bug? -->
59+
- [ ] Yes, I can do this and open a PR for your review.
60+
- [ ] Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this fixed.
61+
- [ ] No, I'd prefer if someone else fixed this. I don't have the time and/or don't know what the root cause of the problem is.
62+

‎.github/ISSUE_TEMPLATE/feature_request.md‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
---
2+
name: Feature request
3+
about: Suggest an idea for this project
4+
title: FEATURE - [your feature request title here]
5+
labels: enhancement
6+
assignees: ''
7+
8+
---
9+
10+
**Are you a Fivetran customer?**
11+
<!--- Please tell us your name, title and company -->
12+
13+
**Is your feature request related to a problem? Please describe.**
14+
<!--- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] -->
15+
16+
**Describe the solution you'd like**
17+
<!--- A clear and concise description of what you want to happen. -->
18+
19+
**Describe alternatives you've considered**
20+
<!--- A clear and concise description of any alternative solutions or features you've considered. -->
21+
22+
**Additional context**
23+
<!--- Add any other context or screenshots about the feature request here. -->
24+
25+
**Please indicate the level of urgency and business impact of this request**
26+
<!--- Is this impacting your reporting of a critical dashboard? What problem would implementing this feature request solve? Provide us context so that we can prioritize accordingly. -->
27+
28+
**Are you interested in contributing to this package?**
29+
<!--- Our packages are all open-source and we welcome contributions from the community! Are you interested in helping us implement this feature? -->
30+
- [ ] Yes, I can do this and open a PR for your review.
31+
- [ ] Possibly, but I'm not quite sure how to do this. I'd be happy to do a live coding session with someone to get this work implemented.
32+
- [ ] No, I'd prefer if someone else did this. I don't have the time and/or don't know how to incorporate the changes necessary.

‎.github/ISSUE_TEMPLATE/question.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
name: Question
3+
about: Confused about how to use the package? Ask us a question
4+
title: QUESTION - [your question title here]
5+
labels: question
6+
assignees: ''
7+
8+
---
9+
10+
**Are you a Fivetran customer?**
11+
<!--- Please tell us your name, title and company -->
12+
13+
**Your Question**
14+
<!--- What is the question you would like to ask us about the package -->
15+
16+
**Additional context**
17+
<!--- Add any other context or screenshots about the feature request here. -->
18+
19+
**Please indicate the level of urgency and business impact of this request**
20+
<!--- Is this question blocking you from completing urgent work? Give us more context to help us prioritize this question. -->

‎.github/pull_request_template.md‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
**Are you a current Fivetran customer?**
2+
<!--- Please tell us your name, title and company -->
3+
4+
**What change(s) does this PR introduce?**
5+
<!--- Describe what changes your PR introduces to the package -->
6+
7+
**Does this PR introduce a breaking change?**
8+
<!--- Does this PR introduce changes that -->
9+
<!--- Mark yes or no (eg. [x] Yes). If yes, link the issue. -->
10+
- [ ] Yes (please provide breaking change details below.)
11+
- [ ] No (please provide explanation how the change is non breaking below.)
12+
13+
**Is this PR in response to a previously created Issue**
14+
<!--- If an Issue was created it is helpful to track the progress by linking it in the PR. -->
15+
<!--- Mark yes or no (eg. [x] Yes). If yes, link the issue. -->
16+
- [ ] Yes, Issue [link issue number here]
17+
- [ ] No
18+
19+
**How did you test the PR changes?**
20+
<!--- Proof of testing is required in order for the PR to be approved. -->
21+
<!--- To check a box, remove the space and insert an x in the box (eg. [x] CircleCi). -->
22+
- [ ] CircleCi <!--- CircleCi testing is only applicable to Fivetran employees. -->
23+
- [ ] Other (please provide additional testing details below)
24+
25+
**Select which warehouse(s) were used to test the PR**
26+
<!--- To check a warehouse remove the space and insert an x in the box (eg. [x] Bigquery). -->
27+
- [ ] BigQuery
28+
- [ ] Redshift
29+
- [ ] Snowflake
30+
- [ ] Postgres
31+
- [ ] Databricks
32+
- [ ] Other (provide details below)
33+
34+
**Provide an emoji that best describes your current mood**
35+
<!--- For a complete list of markdown compatible emojis check our this git repo (https://gist.github.com/rxaviers/7360908) -->
36+
:dancer:
37+
38+
**Feedback**
39+
40+
We are so excited you decided to contribute to the Fivetran community dbt package! We continue to work to improve the packages and would greatly appreciate your [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next.

‎.gitignore‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
2+
target/
3+
dbt_modules/
4+
logs/
5+
.DS_Store

‎README.md‎

Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
[![Apache License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) ![dbt logo and version](https://img.shields.io/static/v1?logo=dbt&label=dbt-version&message=0.20.x&color=orange)
2+
# Pendo (Source)
3+
4+
This package models Pendo data from [Fivetran's connector](https://fivetran.com/docs/applications/pendo). It uses data in the format described by [this ERD](https://fivetran.com/docs/applications/pendo#schemainformation).
5+
6+
This package enriches your Fivetran data by doing the following:
7+
* Adds descriptions to tables and columns that are synced using Fivetran
8+
* Adds column-level testing where applicable. For example, all primary keys are tested for uniqueness and non-null values.
9+
* Models staging tables, which will be used in our transform package
10+
11+
## Models
12+
13+
This package contains staging models, designed to work simultaneously with our [Pendo modeling package](https://github.com/fivetran/dbt_pendo). The staging models:
14+
* Remove any rows that are soft-deleted
15+
* Name columns consistently across all packages:
16+
* Boolean fields are prefixed with `is_` or `has_`
17+
* Timestamps are appended with `_at`
18+
* ID primary keys are prefixed with the name of the table. For example, a user table's ID column is renamed `user_id`
19+
20+
## Installation Instructions
21+
Check [dbt Hub](https://hub.getdbt.com/) for the latest installation instructions, or [read the dbt docs](https://docs.getdbt.com/docs/package-management) for more information on installing packages.
22+
23+
## Configuration
24+
25+
By default, this package looks for your Pendo data in the `pendo` schema of your [target database](https://docs.getdbt.com/docs/running-a-dbt-project/using-the-command-line-interface/configure-your-profile). If this is not where your Pendo data is, add the following configuration to your `dbt_project.yml` file:
26+
27+
```yml
28+
# dbt_project.yml
29+
30+
...
31+
config-version: 2
32+
33+
vars:
34+
pendo_database: your_database_name
35+
pendo_schema: your_schema_name
36+
```
37+
38+
### Passthrough Columns
39+
40+
This package includes all of the source columns that are defined in the macros folder. We recommend including custom columns in this package because the staging models only bring in the standard columns for the `EVENT`, `FEATURE_EVENT`, `PAGE_EVENT`, `ACCOUNT_HISTORY`, and `VISITOR_HISTORY` tables.
41+
42+
You can add more columns using our passthrough column variables. These variables allow the passthrough columns to be aliased (`alias`) and casted (`transform_sql`) if you want, although it is not required. You can configure datatype casting by using a SQL snippet within the `transform_sql` key. You may add the desired SQL snippet while omitting the `as field_name` part of the casting statement - we rename this column with the alias attribute - and your custom passthrough columns will be casted accordingly.
43+
44+
Use the following format for declaring the respective passthrough variables:
45+
46+
```yml
47+
# dbt_project.yml
48+
49+
vars:
50+
51+
pendo__feature_event_pass_through_columns:
52+
- name: "custom_crazy_field_name"
53+
alias: "normal_field_name"
54+
pendo__page_event_pass_through_columns:
55+
- name: "property_field_id"
56+
alias: "new_name_for_this_field_id"
57+
transform_sql: "cast(new_name_for_this_field as int64)"
58+
- name: "this_other_field"
59+
transform_sql: "cast(this_other_field as string)"
60+
pendo__event_pass_through_columns:
61+
- name: "well_named_field_1"
62+
pendo__account_history_pass_through_columns:
63+
- name: "well_named_field_2"
64+
pendo__visitor_history_pass_through_columns:
65+
- name: "well_named_field_3"
66+
```
67+
68+
### Changing the Build Schema
69+
70+
By default, this package builds the Pendo staging models within a schema titled (`<target_schema>` + `_stg_pendo`) in your target database. If this is not where you would like your Pendo staging data to be written to, add the following configuration to your `dbt_project.yml` file:
71+
72+
```yml
73+
# dbt_project.yml
74+
75+
...
76+
models:
77+
pendo_source:
78+
+schema: my_new_schema_name # leave blank for just the target_schema
79+
```
80+
81+
> Note that if your profile does not have permissions to create schemas in your warehouse, you can set the `+schema` to blank. The package will then write all tables to your pre-existing target schema.
82+
83+
## Contributions
84+
85+
Don't see a model or specific metric you would have liked to be included? Notice any bugs when installing
86+
and running the package? If so, we highly encourage and welcome contributions to this package!
87+
Please create issues or open PRs against `master`. Check out [this post](https://discourse.getdbt.com/t/contributing-to-a-dbt-package/657) on the best workflow for contributing to a package.
88+
89+
## Database Support
90+
91+
This package has been tested on BigQuery, Snowflake, Redshift, Postgres, and Databricks.
92+
93+
## Resources:
94+
- Provide [feedback](https://www.surveymonkey.com/r/DQ7K7WW) on our existing dbt packages or what you'd like to see next
95+
- Have questions, feedback, or need help? Book a time during our office hours [using Calendly](https://calendly.com/fivetran-solutions-team/fivetran-solutions-team-office-hours) or email us at solutions@fivetran.com
96+
- Find all of Fivetran's pre-built dbt packages in our [dbt hub](https://hub.getdbt.com/fivetran/)
97+
- Learn how to orchestrate [dbt transformations with Fivetran](https://fivetran.com/docs/transformations/dbt)
98+
- Learn more about Fivetran overall [in our docs](https://fivetran.com/docs)
99+
- Check out [Fivetran's blog](https://fivetran.com/blog)
100+
- Learn more about dbt [in the dbt docs](https://docs.getdbt.com/docs/introduction)
101+
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
102+
- Join the [chat](http://slack.getdbt.com/) on Slack for live discussions and support
103+
- Find [dbt events](https://events.getdbt.com) near you
104+
- Check out [the dbt blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices

‎dbt_project.yml‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
config-version: 2
2+
3+
name: 'pendo_source'
4+
version: '0.1.0'
5+
6+
require-dbt-version: ">=0.20.0"
7+
8+
models:
9+
pendo_source:
10+
tmp:
11+
+materialized: view
12+
+materialized: table
13+
+schema: stg_pendo
14+
15+
vars:
16+
pendo_source:
17+
account_history: "{{ source('pendo', 'account_history') }}"
18+
application_history: "{{ source('pendo', 'application_history') }}"
19+
event: "{{ source('pendo', 'event') }}"
20+
feature_event: "{{ source('pendo', 'feature_event') }}"
21+
feature_history: "{{ source('pendo', 'feature_history') }}"
22+
group: "{{ source('pendo', 'group') }}"
23+
guide_event: "{{ source('pendo', 'guide_event') }}"
24+
guide_history: "{{ source('pendo', 'guide_history') }}"
25+
guide_step_history: "{{ source('pendo', 'guide_step_history') }}"
26+
page_event: "{{ source('pendo', 'page_event') }}"
27+
page_history: "{{ source('pendo', 'page_history') }}"
28+
page_rule_history: "{{ source('pendo', 'page_rule_history') }}"
29+
user: "{{ source('pendo', 'user') }}"
30+
visitor_history: "{{ source('pendo', 'visitor_history') }}"
31+
poll: "{{ source('pendo', 'poll') }}"
32+
poll_event: "{{ source('pendo', 'poll_event') }}"
33+
visitor_account_history: "{{ source('pendo', 'visitor_account_history') }}"
34+
35+
pendo__event_pass_through_columns: []
36+
pendo__feature_event_pass_through_columns: []
37+
pendo__page_event_pass_through_columns: []
38+
pendo__account_history_pass_through_columns: []
39+
pendo__visitor_history_pass_through_columns: []

0 commit comments

Comments
 (0)