Should we export compressed json from bigquery to gcs?

[benhammondmusic]

Compressing NDJSON: BigQuery Export vs Express Middleware

Current Flow

• BigQuery → NDJSON → GCS → Express (compression middleware) → Frontend

Proposal: Compress at Export

Compressing during BigQuery export instead of in Express:

Benefits

• Reduced GCS storage costs
• Lower bandwidth usage across pipeline
• Less Express server CPU load
• Faster response times

Implementation

1. Add compression parameter to BigQuery exporter
2. Update file extensions (add .gz)
3. Modify Express to:
   • Skip compression middleware for these routes
   • Set proper headers (Content-Encoding: gzip)
   • Stream compressed files directly

Considerations

• Browsers handle decompression automatically
• Any direct data processing may need updates

Discussion

• Should we benchmark both approaches?
• Any compatibility issues with downstream consumers?