Merge pull request #41 from DataRecce/document/getting-started-by-jaffle-shop

[Document] Add the Jaffle Shop example

Author: popcornylu

README.md

@@ -36,55 +36,34 @@ jaffle_shop:
 
 ## Getting Started
 
+[5 minutes walkthrough by jaffle shop example](./docs/get-started-jaffle-shop.md)
+
 1. Installation
 
    ```
    pip install recce
    ```
 
-1. Recce use [dbt artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts) to interact with your dbt project. You need to prepare the artifacts for the base environment.
-
-   ```shell
-   # transform the data to data warehouse
-   dbt run --target prod
-
-   # generate the catalog.json
-   dbt docs generate --target prod
-   ```
-
-   The artifacts are generated within the `target/` directory. Copy these artifacts into the `target-base/` directory as the base state to diff.
-
+1. Go to your dbt project
    ```
-   mkdir -p target-base/
-   cp -R target/ target-base/
+   cd your-dbt-project/
    ```
+1. **Prepare base artifacts**: DBT generates [artifacts](https://docs.getdbt.com/reference/artifacts/dbt-artifacts) when every invocation. You can find these files in the `target/` folder.
 
-1. Develop your awesome features
+   | artifacts               | dbt command                |
+   | ----------------------- | -------------------------- |
+   | manifest.json           | `dbt run`, `dbt build`, .. |
+   | catalog.json (optional) | `dbt docs generate`        |
 
-   ```shell
-   # transform the data to data warehouse
-   dbt run
+   Copy the artifacts for base environment to `target-base/`
 
-   # generate the catalog.json
-   dbt docs generate
-   ```
-
-1. Run the recce server
+1. Run the recce server.
 
    ```
    recce server
    ```
 
-   and open the url link
-
-1. Check the lineage diff to see the modified node. Click one node to see the schema difference.
-1. Switch to **query** tab, Write and run a query diff. It would query on the both side and diff the query results.
-
-   ```jinja
-   select * from {{ ref("mymodel") }}
-   ```
-
-   where `ref` is a Jinja function to reference a model name.
+   Recce would diff environments between `target/` and `target-base/`
 
 ## Query Diff
 

docs/assets/jaffle_shop_lineage.png

@@ -0,0 +1,93 @@
+## Getting Started by Jaffle Shop
+
+**Jaffle Shop** is an example project officially provided by dbt-labs. This document uses [jaffle_shop_duckdb](https://github.com/dbt-labs/jaffle_shop_duckdb) to enable you to start using **recce** locally from scratch within five minutes.
+
+1. Clone the “Jaffle Shop” dbt data project
+
+   ```bash
+   git clone git@github.com:dbt-labs/jaffle_shop_duckdb.git
+   cd jaffle_shop_duckdb
+   ```
+
+1. Prepare virtual env
+   ```bash
+   python -m venv venv
+   source venv/bin/activate
+   ```
+1. Installation
+   ```bash
+   pip install -r requirements.txt
+   pip install recce
+   ```
+1. Provide additional environment to compare. Edit `./profiles.yml` to add one more target.
+
+   ```diff
+   jaffle_shop:
+       target: dev
+       outputs:
+       dev:
+           type: duckdb
+           path: 'jaffle_shop.duckdb'
+           threads: 24
+   +   prod:
+   +     type: duckdb
+   +     path: 'jaffle_shop.duckdb'
+   +     schema: prod
+   +     threads: 24
+   ```
+
+1. Prepare production environment
+
+   ```bash
+   dbt build --target prod
+   dbt docs generate --target prod
+   ```
+
+   and copy the artifacts to `./target-base/`
+
+   ```bash
+   cp -R target/ target-base/
+   ```
+
+1. Prepare development environment. First, edit an existing model `./models/staging/stg_payments.sql`.
+
+   ```diff
+   ...
+
+   renamed as (
+            payment_method,
+
+   -        -- `amount` is currently stored in cents, so we convert it to dollars
+   -        amount / 100 as amount
+   +        amount
+
+            from source
+   )
+   ```
+
+   run on development environment.
+
+   ```bash
+   dbt build
+   dbt docs generate
+   ```
+
+1. Run the recce server
+
+   ```bash
+   recce server
+   ```
+
+   Open the link http://0.0.0.0:8000, you can see the lineage diff
+
+   ![](assets/jaffle_shop_lineage.png)
+
+1. Switch to the **Query** tab, run this query
+
+   ```sql
+   select * from {{ ref("orders") }} order by 1
+   ```
+
+   Click on the 🔑 icon next to the `order_id` column to compare records that are uniquely identified by their `order_id`.
+
+   ![](assets/jaffle_shop_query.png)