Skip to content

Fix Unicode character handling in API responses #44

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

solavrc
Copy link

@solavrc solavrc commented Mar 25, 2025

Problem

When the backend returns responses containing Unicode characters (like Japanese), the Looker proxy encounters character encoding issues and returns 500 errors to the frontend.

Solution

This PR implements a base64 encoding/decoding approach to safely transmit Unicode characters:

  • Backend: Encode JSON responses as base64 strings before sending them through the Looker proxy
  • Frontend: Add a utility function that decodes base64 responses with proper UTF-8 handling
  • Ensure multi-byte characters are correctly processed using TextDecoder

Implementation Details

  1. Modified backend endpoint handlers to encode responses using Buffer.from(JSON.stringify()).toString('base64')
  2. Created base64Helper.ts utility for frontend to decode responses with Unicode support
  3. Updated API utility functions to use the base64 decoder for all responses

This approach ensures consistent handling of all character sets, including Japanese, without relying on Looker proxy's character encoding capabilities.

@solavrc solavrc requested a review from a team as a code owner March 25, 2025 03:52
Copy link

google-cla bot commented Mar 25, 2025

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

Comment on lines +16 to +19
// Convert base64 to binary data using the approach from MDN
// https://developer.mozilla.org/en-US/docs/Web/API/Window/btoa#unicode_strings
const binString = atob(base64String);
const bytes = Uint8Array.from(binString, (m) => m.charCodeAt(0));
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The webpack configuration explicitly disables Buffer polyfill with:

resolve: {
extensions: [".tsx", ".ts", ".js"],
fallback: { buffer: false },

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant