Skip to content

Commit ed34bd6

Browse files
cabljaccabljac
authored
docs(firestore-bigquery-export): provided bash/ps1 scripts for granting cross-project access to the extension (#2241)
1 parent f87b847 commit ed34bd6

File tree

5 files changed

+210
-5
lines changed

5 files changed

+210
-5
lines changed

‎firestore-bigquery-export/POSTINSTALL.md

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -54,13 +54,44 @@ Enabling wildcard references will provide an additional STRING based column. The
5454
5555
`Clustering` will not need to create or modify a table when adding clustering options, this will be updated automatically.
5656
57-
### Configuring Cross-Platform BigQuery Setup
57+
#### Cross-project Streaming
5858
59-
When defining a specific BigQuery project ID, a manual step to set up permissions is required:
59+
By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this:
6060
61-
1. Navigate to https://console.cloud.google.com/iam-admin/iam?project=${param:BIGQUERY_PROJECT_ID}
62-
2. Add the **BigQuery Data Editor** role to the following service account:
63-
`ext-${param:EXT_INSTANCE_ID}@${param:PROJECT_ID}.iam.gserviceaccount.com`.
61+
1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID.
62+
63+
2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts:
64+
65+
**For Linux/Mac (Bash):**
66+
```bash
67+
curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh
68+
chmod +x grant-crossproject-access.sh
69+
./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID]
70+
```
71+
72+
**For Windows (PowerShell):**
73+
```powershell
74+
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1"
75+
.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID]
76+
```
77+
78+
**Parameters:**
79+
For Bash script:
80+
- `-f`: Your Firebase (source) project ID
81+
- `-b`: Your target BigQuery project ID
82+
- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"
83+
84+
For PowerShell script:
85+
- `-FirebaseProject`: Your Firebase (source) project ID
86+
- `-BigQueryProject`: Your target BigQuery project ID
87+
- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"
88+
89+
**Prerequisites:**
90+
- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured
91+
- You must have permission to grant IAM roles on the target BigQuery project
92+
- The extension must be installed before running the script
93+
94+
**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.
6495

6596
### _(Optional)_ Import existing documents
6697

‎firestore-bigquery-export/PREINSTALL.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -100,6 +100,45 @@ If you follow these steps, your changelog table should be created using your cus
100100

101101
After your data is in BigQuery, you can run the [schema-views script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md) (provided by this extension) to create views that make it easier to query relevant data. You only need to provide a JSON schema file that describes your data structure, and the schema-views script will create the views.
102102

103+
#### Cross-project Streaming
104+
105+
By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this:
106+
107+
1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID.
108+
109+
2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts:
110+
111+
**For Linux/Mac (Bash):**
112+
```bash
113+
curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh
114+
chmod +x grant-crossproject-access.sh
115+
./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID]
116+
```
117+
118+
**For Windows (PowerShell):**
119+
```powershell
120+
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1"
121+
.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID]
122+
```
123+
124+
**Parameters:**
125+
For Bash script:
126+
- `-f`: Your Firebase (source) project ID
127+
- `-b`: Your target BigQuery project ID
128+
- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"
129+
130+
For PowerShell script:
131+
- `-FirebaseProject`: Your Firebase (source) project ID
132+
- `-BigQueryProject`: Your target BigQuery project ID
133+
- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"
134+
135+
**Prerequisites:**
136+
- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured
137+
- You must have permission to grant IAM roles on the target BigQuery project
138+
- The extension must be installed before running the script
139+
140+
**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.
141+
103142
#### Billing
104143
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)
105144

‎firestore-bigquery-export/README.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -108,6 +108,45 @@ If you follow these steps, your changelog table should be created using your cus
108108

109109
After your data is in BigQuery, you can run the [schema-views script](https://github.com/firebase/extensions/blob/master/firestore-bigquery-export/guides/GENERATE_SCHEMA_VIEWS.md) (provided by this extension) to create views that make it easier to query relevant data. You only need to provide a JSON schema file that describes your data structure, and the schema-views script will create the views.
110110

111+
#### Cross-project Streaming
112+
113+
By default, the extension exports data to BigQuery in the same project as your Firebase project. However, you can configure it to export to a BigQuery instance in a different Google Cloud project. To do this:
114+
115+
1. During installation, set the `BIGQUERY_PROJECT_ID` parameter to your target BigQuery project ID.
116+
117+
2. After installation, you'll need to grant the extension's service account the necessary BigQuery permissions on the target project. You can use our provided scripts:
118+
119+
**For Linux/Mac (Bash):**
120+
```bash
121+
curl -O https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.sh
122+
chmod +x grant-crossproject-access.sh
123+
./grant-crossproject-access.sh -f SOURCE_FIREBASE_PROJECT -b TARGET_BIGQUERY_PROJECT [-i EXTENSION_INSTANCE_ID]
124+
```
125+
126+
**For Windows (PowerShell):**
127+
```powershell
128+
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/firebase/extensions/master/firestore-bigquery-export/scripts/grant-crossproject-access.ps1" -OutFile "grant-crossproject-access.ps1"
129+
.\grant-crossproject-access.ps1 -FirebaseProject SOURCE_FIREBASE_PROJECT -BigQueryProject TARGET_BIGQUERY_PROJECT [-ExtensionInstanceId EXTENSION_INSTANCE_ID]
130+
```
131+
132+
**Parameters:**
133+
For Bash script:
134+
- `-f`: Your Firebase (source) project ID
135+
- `-b`: Your target BigQuery project ID
136+
- `-i`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"
137+
138+
For PowerShell script:
139+
- `-FirebaseProject`: Your Firebase (source) project ID
140+
- `-BigQueryProject`: Your target BigQuery project ID
141+
- `-ExtensionInstanceId`: (Optional) Extension instance ID if different from default "firestore-bigquery-export"
142+
143+
**Prerequisites:**
144+
- You must have the [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and configured
145+
- You must have permission to grant IAM roles on the target BigQuery project
146+
- The extension must be installed before running the script
147+
148+
**Note:** If extension installation is failing to create a dataset on the target project initially due to missing permissions, don't worry. The extension will automatically retry once you've granted the necessary permissions using these scripts.
149+
111150
#### Billing
112151
To install an extension, your project must be on the [Blaze (pay as you go) plan](https://firebase.google.com/pricing)
113152

‎firestore-bigquery-export/scripts/grant-crossproject-access.ps1

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Help message
2+
function Show-Help {
3+
Write-Host "Usage: .\grant-crossproject-access.ps1 -FirebaseProject <PROJECT_ID> -BigQueryProject <PROJECT_ID> [-ExtensionInstanceId <INSTANCE_ID>]"
4+
Write-Host
5+
Write-Host "Parameters:"
6+
Write-Host " -FirebaseProject Firebase (source) project ID"
7+
Write-Host " -BigQueryProject BigQuery project ID where dataset will be created"
8+
Write-Host " -ExtensionInstanceId Extension instance ID (default: firestore-bigquery-export)"
9+
exit 1
10+
}
11+
12+
# Parameters
13+
param(
14+
[Parameter(Mandatory=$true)]
15+
[string]$FirebaseProject,
16+
17+
[Parameter(Mandatory=$true)]
18+
[string]$BigQueryProject,
19+
20+
[Parameter(Mandatory=$false)]
21+
[string]$ExtensionInstanceId = "firestore-bigquery-export"
22+
)
23+
24+
# Construct service account email
25+
$ServiceAccount = "ext-${ExtensionInstanceId}@${FirebaseProject}.iam.gserviceaccount.com"
26+
27+
Write-Host "Using service account: $ServiceAccount"
28+
Write-Host "Adding BigQuery permissions to $ServiceAccount on project: $BigQueryProject"
29+
30+
$confirmation = Read-Host "Continue? (y/N)"
31+
if ($confirmation -notmatch '^[yY]$') {
32+
exit 1
33+
}
34+
35+
# Grant bigquery.dataEditor role
36+
gcloud projects add-iam-policy-binding $BigQueryProject `
37+
--member="serviceAccount:$ServiceAccount" `
38+
--role="roles/bigquery.dataEditor"
39+
40+
# Grant bigquery.dataOwner which includes dataset.create permission
41+
gcloud projects add-iam-policy-binding $BigQueryProject `
42+
--member="serviceAccount:$ServiceAccount" `
43+
--role="roles/bigquery.dataOwner"

‎firestore-bigquery-export/scripts/grant-crossproject-access.sh

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
#!/bin/bash
2+
3+
# Help message
4+
function show_help {
5+
echo "Usage: $0 -f FIREBASE_PROJECT -b BIGQUERY_PROJECT -i EXTENSION_INSTANCE_ID"
6+
echo
7+
echo "Options:"
8+
echo " -f Firebase (source) project ID"
9+
echo " -b BigQuery project ID where dataset will be created"
10+
echo " -i Extension instance ID (default: firestore-bigquery-export)"
11+
echo " -h Show this help message"
12+
exit 1
13+
}
14+
15+
# Set default extension instance ID
16+
EXT_INSTANCE_ID="firestore-bigquery-export"
17+
18+
# Parse command line arguments
19+
while getopts "f:b:i:h" opt; do
20+
case $opt in
21+
f) FIREBASE_PROJECT="$OPTARG";;
22+
b) BIGQUERY_PROJECT="$OPTARG";;
23+
i) EXT_INSTANCE_ID="$OPTARG";;
24+
h) show_help;;
25+
?) show_help;;
26+
esac
27+
done
28+
29+
# Check if required arguments are provided
30+
if [ -z "$FIREBASE_PROJECT" ] || [ -z "$BIGQUERY_PROJECT" ]; then
31+
echo "Error: Both Firebase and BigQuery project IDs are required"
32+
show_help
33+
fi
34+
35+
# Construct service account email
36+
SERVICE_ACCOUNT="ext-${EXT_INSTANCE_ID}@${FIREBASE_PROJECT}.iam.gserviceaccount.com"
37+
38+
echo "Using service account: $SERVICE_ACCOUNT"
39+
echo "Adding BigQuery permissions to $SERVICE_ACCOUNT on project: $BIGQUERY_PROJECT"
40+
read -p "Continue? (y/N) " -n 1 -r
41+
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
42+
exit 1
43+
fi
44+
45+
# Grant bigquery.dataEditor role
46+
gcloud projects add-iam-policy-binding $BIGQUERY_PROJECT \
47+
--member="serviceAccount:$SERVICE_ACCOUNT" \
48+
--role="roles/bigquery.dataEditor"
49+
50+
# Grant bigquery.dataOwner which includes dataset.create permission
51+
gcloud projects add-iam-policy-binding $BIGQUERY_PROJECT \
52+
--member="serviceAccount:$SERVICE_ACCOUNT" \
53+
--role="roles/bigquery.dataOwner"

0 commit comments

Comments
 (0)