Bug Report: UTF-8 Character Encoding Issue in Identity Tokenization

[H3L70N]

Bug Report: UTF-8 Character Encoding Issue in Identity Tokenization

Summary

The identity tokenization feature in BLNK fails when processing names containing UTF-8 special characters, producing invalid byte sequences that cannot be stored in the PostgreSQL database.

Environment

• BLNK Version: 1.7.3
• Database: PostgreSQL with UTF-8 encoding
• Deployment: Docker Compose
• Database Encoding: Both server_encoding and client_encoding confirmed as UTF8

Bug Description

When attempting to tokenize an identity that contains UTF-8 special characters (accented characters, umlauts, etc.), the tokenization process corrupts the character encoding, resulting in invalid UTF-8 byte sequences that cause PostgreSQL insertion failures.

Steps to Reproduce

1. Create an identity with a name containing UTF-8 special characters:

   curl -X POST http://your-blnk-instance/identities \
     -H "Content-Type: application/json; charset=utf-8" \
     -d '{"first_name": "Test", "last_name": "Taizé"}'
   

2. Attempt to tokenize the created identity:

   curl -X POST http://your-blnk-instance/identities/{identity_id}/tokenize
   

3. Observe the tokenization failure with UTF-8 encoding error

Expected Behavior

• Identity tokenization should succeed regardless of UTF-8 characters in the data
• Tokenized values should maintain valid UTF-8 encoding
• Special characters should be preserved through the tokenization process

Actual Behavior

• Tokenization fails with PostgreSQL error: pq: invalid byte sequence for encoding "UTF8"
• Identity creation succeeds (201 status), but tokenization fails (400 status)
• Invalid byte sequences are generated during the tokenization process

Test Cases and Results

Input Character	Original Name	Invalid Byte Sequence	Status
é	"Taizé"	0xe9 0x3a 0x6f	❌ Fails
ç (assumed)	[Test name]	0xe7 0x7a 0x6a	❌ Fails
ü (assumed)	[Test name]	0xfc	❌ Fails
Ö (assumed)	[Test name]	0xd6 0x78	❌ Fails

Error Logs

server  | [GIN] 2025/07/29 - 08:55:17 | 400 |     4.40429ms |    196.22.51.22 | POST     "/identities/idt_36ec0643-5e63-4c07-88c8-027e244feb8c/tokenize"
server  | time="2025-07-29T08:55:17Z" level=error msg="pq: invalid byte sequence for encoding \"UTF8\": 0xe9 0x3a 0x6f"
server  | [GIN] 2025/07/29 - 09:17:37 | 201 |   11.921687ms |   197.235.78.78 | POST     "/identities"

server  | time="2025-07-29T09:17:37Z" level=info msg="2025/07/29 09:17:37  [*] Successfully enqueued index data: idt_a52f1671-dd2f-4a33-950a-4e5dc93c084c"

server  | time="2025-07-29T09:17:37Z" level=error msg="pq: invalid byte sequence for encoding "UTF8": 0xe7 0x7a 0x6a"

server  | [GIN] 2025/07/29 - 09:17:37 | 400 |    5.033152ms |   197.235.78.78 | POST     "/identities/idt_a52f1671-dd2f-4a33-950a-4e5dc93c084c/tokenize"
server  | time="2025-07-29T09:18:33Z" level=error msg="pq: invalid byte sequence for encoding "UTF8": 0xfc"

server  | time="2025-07-29T09:18:40Z" level=error msg="pq: invalid byte sequence for encoding "UTF8": 0xd6 0x78"

Database State Verification

Database query showing successful storage of original UTF-8 data but missing tokenization:

blnk=# select id, first_name, last_name from identity order by created_at desc limit 4;
 id |                         first_name                          |                          last_name
----+-------------------------------------------------------------+-------------------------------------------------------------
 91 | FPT:Wmkpe:8+ZqvjTMyiXzO61ZZZqF4IF6TY9/P4lebcMjZj68omPm      | Taizé
 90 | FPT:Wmkpe:F6A11I3MpqmS/J3iYCSGfM+tvFQ4NZoAq/N7Rb+un7rJ      | Taizé
 89 | FPT:Gmhb:nZixc8T/L4hz8LWSyqZwf6pOYvJe4Lxrnmzo0qKAeSo=       | FPT:Jjr:cb66HoZNB1f6EdUtdjQlya3ylWAkCBLZNkQV07Phpg==
 88 | FPT:NVCWCR:jdBak5hLrim3TixylJrvHvqf5gnuwPz1a+fJFdd4CbYIvw== | FPT:FWAFFH:4wnqalDMf12aljEqDLkrVk1ZKlg7T1HE0yBRGLvZ/hmImg==

Note: Entries 88 and 89 have properly tokenized last_name fields, while entries 90 and 91 show the raw "Taizé" value, indicating tokenization failure.

Root Cause Analysis

The issue appears to be in the tokenization algorithm itself, which:

1. ✅ Correctly reads UTF-8 data from the database
2. ❌ Corrupts UTF-8 multi-byte sequences during the encryption/tokenization process
3. ❌ Produces invalid UTF-8 byte sequences as output
4. ❌ Fails when attempting to store the corrupted result back to PostgreSQL

The tokenization process seems to be treating UTF-8 multi-byte characters as individual single-byte characters, breaking the UTF-8 encoding specification.

Impact

• Severity: High - Affects international users with non-ASCII names
• Scope: All identity tokenization operations involving UTF-8 special characters
• Workaround: Currently limited to ASCII-only characters in tokenized fields

Suggested Fix

The tokenization implementation needs to:

1. Properly handle UTF-8 string input
2. Ensure the encryption/tokenization process preserves UTF-8 encoding
3. Validate that tokenized output maintains valid UTF-8 sequences
4. Add comprehensive UTF-8 character testing to the test suite

Test Recommendations

Add test cases covering:

• Common accented characters (é, è, à, ñ, ç)
• Umlauts and special European characters (ü, ö, ä, ß)
• Unicode characters from various languages
• Edge cases with combining characters and multi-byte sequences

Additional Information

• Database encoding is correctly configured as UTF-8
• Identity creation works perfectly with UTF-8 characters
• Only the tokenization step fails
• Issue is reproducible across different UTF-8 special characters

---

Reporter: Helton Guambe @pavulla-tech
Date: July 29, 2025
Priority: High

[jerry-enebeli]

I'll look into this, @H3L70N. Thanks for reporting this!

[jerry-enebeli]

@H3L70N just published v0.10.8 with the fix. Thanks for the insight.