Skip to content

Commit 286ce14

Browse files
authored
docs(firestore-bigquery-export): add documentation on reconfig workaround (#2250)
1 parent c14805d commit 286ce14

File tree

3 files changed

+40
-0
lines changed

3 files changed

+40
-0
lines changed

firestore-bigquery-export/CHANGELOG.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ docs - improve cross-project IAM documentation
88

99
fix - emit correct events to extension, backwardly compatible.
1010

11+
docs - add documentation on workarounds to mitigate data loss during extension updates
12+
1113
## Version 0.1.56
1214

1315
feat - improve sync strategy by immediately writing to BQ, and using cloud tasks only as a last resort

firestore-bigquery-export/PREINSTALL.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -214,6 +214,25 @@ For PowerShell script:
214214

215215
**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.
216216

217+
#### Mitigating Data Loss During Extension Updates
218+
219+
When updating or reconfiguring this extension, there may be a brief period where data streaming from Firestore to BigQuery is interrupted. While this limitation exists within the Extensions platform, we provide two strategies to mitigate potential data loss.
220+
221+
##### Strategy 1: Post-Update Import
222+
After reconfiguring the extension, run the import script on your collection to ensure all data is captured. Refer to the "Import Existing Documents" section above for detailed steps.
223+
224+
##### Strategy 2: Parallel Instance Method
225+
1. Install a second instance of the extension that streams to a new BigQuery table
226+
2. Reconfigure the original extension
227+
3. Once the original extension is properly configured and streaming events
228+
4. Uninstall the second instance
229+
5. Run a BigQuery merge job to combine the data from both tables
230+
231+
##### Considerations
232+
- Strategy 1 is simpler but may result in duplicate records that need to be deduplicated
233+
- Strategy 2 requires more setup but provides better data continuity
234+
- Choose the strategy that best aligns with your data consistency requirements and operational constraints
235+
217236
#### Billing
218237
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)
219238

firestore-bigquery-export/README.md

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -222,6 +222,25 @@ For PowerShell script:
222222

223223
**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.
224224

225+
#### Mitigating Data Loss During Extension Updates
226+
227+
When updating or reconfiguring this extension, there may be a brief period where data streaming from Firestore to BigQuery is interrupted. While this limitation exists within the Extensions platform, we provide two strategies to mitigate potential data loss.
228+
229+
##### Strategy 1: Post-Update Import
230+
After reconfiguring the extension, run the import script on your collection to ensure all data is captured. Refer to the "Import Existing Documents" section above for detailed steps.
231+
232+
##### Strategy 2: Parallel Instance Method
233+
1. Install a second instance of the extension that streams to a new BigQuery table
234+
2. Reconfigure the original extension
235+
3. Once the original extension is properly configured and streaming events
236+
4. Uninstall the second instance
237+
5. Run a BigQuery merge job to combine the data from both tables
238+
239+
##### Considerations
240+
- Strategy 1 is simpler but may result in duplicate records that need to be deduplicated
241+
- Strategy 2 requires more setup but provides better data continuity
242+
- Choose the strategy that best aligns with your data consistency requirements and operational constraints
243+
225244
#### Billing
226245
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)
227246

0 commit comments

Comments
 (0)