You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
- add info that you install the script via npm
- add info about needing Node.js
- add intro line about which extension the script is for
- general style updates
Copy file name to clipboardExpand all lines: firestore-bigquery-export/guides/IMPORT_EXISTING_DOCUMENTS.md
+12-4Lines changed: 12 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,5 @@
1
+
The `fs-bq-import-collection` script is for use with the official Firebase Extension [**Export Collections to BiqQuery**](https://github.com/firebase/extensions/tree/master/firestore-bigquery-export).
2
+
1
3
### Overview
2
4
3
5
The import script (`fs-bq-import-collection`) can read all existing documents in a Cloud Firestore collection and insert them into the raw changelog table created by the Export Collections to BigQuery extension. The script adds a special changelog for each document with the operation of `IMPORT` and the timestamp of epoch. This ensures that any operation on an imported document supersedes the import record.
@@ -6,22 +8,28 @@ You may pause and resume the script from the last batch at any point.
6
8
7
9
#### Important notes
8
10
9
-
+Run the script over the entire collection **_after_** installing the Export Collections to BigQuery extension; otherwise the writes to your database during the import might not be exported to the dataset.
11
+
+You must run the script over the entire collection **_after_** installing the Export Collections to BigQuery extension; otherwise the writes to your database during the import might not be exported to the dataset.
10
12
+ The import script can take up to _O(collection size)_ time to finish. If your collection is large, you might want to consider [loading data from a Cloud Firestore export into BigQuery](https://cloud.google.com/bigquery/docs/loading-data-cloud-firestore).
11
-
+ You will see redundant rows in your raw changelog table:
13
+
+ You will see redundant rows in your raw changelog table if either of the following happen:
12
14
13
15
+ If document changes occur in the time between installing the extension and running this import script.
14
16
+ If you run the import script multiple times over the same collection.
15
17
16
18
### Install and run the script
17
19
18
-
This import script uses several values from your installation of the extension:
20
+
Install the import script using `npm` (the Node Package Manager). Note that the import script uses several values from your installation of the extension:
19
21
20
22
+`${PROJECT_ID}`: the project ID for the Firebase project in which you installed the extension
21
23
+`${COLLECTION_PATH}`: the collection path that you specified during extension installation
22
24
+`${DATASET_ID}`: the ID that you specified for your dataset during extension installation
23
25
24
-
1. Run `npx @firebaseextensions/fs-bq-import-collection`.
26
+
1. Make sure that you have [Node.js](https://www.nodejs.org/) installed in order to access the `npm` command tools.
27
+
28
+
1. Install the import script via `npm` by running the following command:
29
+
30
+
```
31
+
npx @firebaseextensions/fs-bq-import-collection
32
+
```
25
33
26
34
1. When prompted, enter the Cloud Firestore collection path that you specified during extension installation, `${COLLECTION_PATH}`.
0 commit comments