No Handling of 4-Byte UTF-16 Characters During UTF-8 Conversion on Windows

[LXY1226]

Description:

When I attempted to integrate zpaq via cmd in a Go program, I encountered an issue while using zpaq l -checksum to verify if files were written correctly. Upon inspecting the output of zpaq, I noticed the presence of non-existent filenames. After investigation, I found that the UTF-8 output unexpectedly contained 4-byte UTF-16 sequences.

Below is my fix. By converting UTF-8 to UTF-16 and then back to UTF-8 using the correct conversion function, I successfully resolved the issue in my case:

std::string wtou(const std::wstring& w) {
	if (w.empty()) return std::string();
	int size = WideCharToMultiByte(CP_UTF8, 0, w.c_str(), -1, nullptr, 0, nullptr, nullptr);
	std::string result(size - 1, 0);
	WideCharToMultiByte(CP_UTF8, 0, w.c_str(), -1, &result[0], size, nullptr, nullptr);
	return result;
}

void printUTF8(const char *s, FILE *f = stdout) {
	assert(f);
	assert(s);
	if (flagsilent)
		return;
#ifdef unix
	fprintf(f, "%s", s);
#else
	const HANDLE h = (HANDLE) _get_osfhandle(_fileno(f));
	DWORD ft = GetFileType(h);
	std::wstring w = utow(s, '/');  // <<<<<
	std::string utf8_str = wtou(w); // <<<<<
	if (ft == FILE_TYPE_CHAR) {
		fflush(f);
		DWORD n = 0;
		WriteConsole(h, w.c_str(), w.size(), &n, 0);
		if (g_output_handle != 0) {
			fprintf(g_output_handle, "%s", utf8_str.c_str());
		}
	} else
        fprintf(f, "%s", utf8_str.c_str());
#endif
}

Impact Path:

When archiving, converting Windows-provided UTF-16 filenames to UTF-8 may omit 4-byte UTF-16 characters, saving them as mixed UTF-16 and UTF-8 strings. This behavior does not cause issues when decompressing on Windows, as it converts back to UTF-16. However, it may cause problems on other platforms.

Test Byte Sequences:

• ☝ (U+261D) - Correctly encoded as 3-byte UTF-8
• 🤔 (U+1F914) - Incorrectly encoded as 4-byte UTF-8

AI-Generated Analysis

Correct UTF-8 Sequence:

F0 9F A4 94  // 🤔 emoji's correct UTF-8 encoding

The Sequence You Got:

ED A0 BE ED B4 94  // This is the result of incorrectly encoding UTF-16 surrogate pairs as UTF-8

Root Cause

This is a typical UTF-16 to UTF-8 conversion error:

1. The Unicode code point for 🤔 is U+1F914.
2. In UTF-16, this character is represented using surrogate pairs: D83E DD14.
3. zpaq might first convert the character to UTF-16 and then incorrectly treat the surrogate pairs as independent Unicode characters when encoding to UTF-8:
   • U+D83E → ED A0 BE (UTF-8)
   • U+DD14 → ED B4 94 (UTF-8)

Apologies for the lack of further explanation. I am currently trying to rescue data that is about to be destroyed by OneDrive. If you need more detailed information, please let me know.

I'm not a native English speaker, so the explanation might seem a bit confusing.

[fcorbelli]

It's an interesting issue

Typically, I don't have file names as odd as that; usually, I’ve conducted tests on the alphabets used by various Eastern countries

This one https://encode.su/threads/3629-Help-from-users-with-non-latin-charset-(China-Russia-the-stranger-the-better)

If you have any test files (I mean with the file names), please attach them to this thread so I can study them.

Thanks for the heads‑up

I'm not a native English speaker, so the explanation might seem a bit confusing.
Neither me 😅

[LXY1226]

this time the "victim" isn’t those strange characters, but the emojis :(

I have created some test files and corresponding outputs:

• test_☝and🤔.txt is a file that should be correctly packed with its filename works correctly in my environment in content.
• test.zpaq is the archive generated using the command zpaqfranz a ../test.zpaq ./ -longpath.
• original_listing.log contains the output from the current (unfixed) version. zpaqfranz l .\test.zpaq -out original_listing.log
• fixed_listing.log contains the output from my modified version. zpaqfranz_fixed l .\test.zpaq -out fixed_listing.log

issue.zip

I have packaged these files into a zip file. I hope these files can help with further analysis and resolving the issue.

[baka-gourd]

I'm having the same problem. It's actually a pretty classic problem. The current function doesn't seem to be able to handle characters after U+FFF0.

U+2E80-U+4E00 includes some of the cjk characters, these are represented as three bytes in utf-8, so they pass the test properly.

However, in fact U+20000-U+31350 are also used to represent cjk characters, with emoji instead being the most difficult character to encounter.

utf-8, as a variable-length encoding, can theoretically be encoded infinitely, but for historical reasons, it only needs to be taken into account up to 0x10ffff, which is the upper limit of utf-16. utf-16 is divided into two- and four-byte bytes, which contain the Surrogate Pair (which are more complicated but will be well documented)

If you need some test text, then the following is it:

𠅁𠅂𠅃𠅄𠅅𠅆𠅇𠅈𠅉𠅊𠅋𠅏𠅌𠅎𠅒

𒀀𒀁𒀂𒀅𒀃𒀄𒀆

𐊠𐊡𐊢𐊣𐊤𐊥𐊦𐊧𐊨𐊱𐊪𐊴𐊵

To be honest this is a problem with c/cpp designed for ascii, other newer languages generally use utf-8/utf-16 as the default representation, so it's rare to have this problem because there are specialized encoders to solve these complex unicode conversions

unipad: https://github.com/Ryosuke839/UnicodePad

https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/

[fcorbelli]

Actually, the issue is much more complex, because these are not just generic strings — they are filenames, which means they are subject to all the rules of the filesystem.

Don't confuse processing arbitrary strings with handling filenames.

It's even worse across different filesystems, each of which has its own internal representation.

On top of that, there's the performance issue: handling millions of strings introduces significant slowdowns, while everyone expects an archiver to be fast.

Therefore, library functions from other languages — though robust — are often far too slow and not the ideal choice.

Right now, I'm implementing remote execution of control code over SSH. Once that's done, I'll take a look at the emoji issue

[baka-gourd]

Actually, the issue is much more complex, because these are not just generic strings — they are filenames, which means they are subject to all the rules of the filesystem.

Don't confuse processing arbitrary strings with handling filenames.

This is reasonable, but given that essentially all modern operating systems are using utf-16 as their internal encoding, it's very bad news. Because this in fact allows all characters to be used as filenames except for the forced exclusion of characters like \

https://learn.microsoft.com/en-us/windows/win32/intl/character-sets-used-in-file-names

As for linux, it's a bit more complicated, depending on the user's choice of locale, but for the most part it's also full utf-8

So, it's a very bad scenario

[fcorbelli]

Well, in fact no
For example, you cannot get a "com1" file on Windows
Or lpt1

Nobody really cares, or at least I don't, on using emoji or whatever as a file name
Ensuring backward compatibility is critical

[baka-gourd]

Well, in fact no
For example, you cannot get a "com1" file on Windows
Or lpt1

Nobody really cares, or at least I don't, on using emoji or whatever as a file name
Ensuring backward compatibility is critical

Maybe some related works, but I haven't test this.

https://arxiv.org/abs/2109.10433

May like -blake3 and -blake3b, as different variants but not turned on by default

[fcorbelli]

You do not have SIMD on, for example, Annapurna-based cheap NASes

And zpaqfranz does run even there 🙂

Remember: everything is so easy, if you use ONE platform or some "just-like" platforms
Even rsync cannot handle filenames in every single case (ex. freebsd vs macos)

[baka-gourd]

https://github.com/simdutf/simdutf

seems this library is widely used in high performance sense, such as browser, bun or haskell

and have single header and correctly fallback when running on Non-SIMD devices

I think I can give it a try. The performance should not be too bad.

[baka-gourd]

speed-test.zip

dataset: https://github.com/lemire/unicode_lipsum

compile command: g++ -O3 -std=c++17 utf_benchmark(_no_simd).cpp -o b.exe

default (with platform optimize):

franz UTF16(u16)->UTF8 (custom_u16_to_utf8)  97.576 us/ts   interp: 418.48 M unit/s
simdutf UTF16(u16)->UTF8  13.902 us/ts   interp: 2937.27 M unit/s
  ratio: 7.02x

franz UTF8->UTF16(u16) (custom_utf8_to_u16)  75.968 us/ts   interp: 1024.22 M unit/s
simdutf UTF8->UTF16(u16)  26.656 us/ts   interp: 2918.97 M unit/s
  ratio: 2.85x

fallback (scalar):

franz UTF16(u16)->UTF8 (custom_u16_to_utf8)  90.276 us/ts   interp: 452.32 M unit/s
simdutf UTF16(u16)->UTF8 (fallback)   71.447 us/ts   interp: 571.52 M unit/s
  ratio: 1.26x

franz UTF8->UTF16(u16) (custom_utf8_to_u16)  107.620 us/ts   interp: 722.98 M unit/s
simdutf UTF8->UTF16(u16) (fallback)   132.451 us/ts   interp: 587.45 M unit/s
  ratio: 0.81x

This is a simple and informal test. When SIMD is enabled, simdutf completely outperforms the custom method. Even when completely fallback to scalar UTF-8 encoding, it still outperforms the custom method, but there is a slight performance drop when encoding UTF-16. simdutf can handle all points completely, which I think is an acceptable result.

Compatibility does need to be considered, so this can be implemented as a switch that is disabled by default for several versions.

I hope this information is helpful to you.

[fcorbelli]

I am unable to replicate

[fcorbelli]

If I understand correctly, the issue concerns only -out, that is, the logging into a possible log file

[LXY1226]

Yes, when using Windows and console direct output, it seems like there is no problem. Let me explain what is going on.

When outputting to the Windows console, it will re-encode UTF-8 back to UTF-16 and left UTF-16 mixed in UTF-8 as-is. This results in "correct" UTF-16 output.

However, when redirecting stdout to a file (pipe) or using the -out option, the above conversion does not occur, resulting in mixed UTF-8 and UTF-16 output.

All of this is because the filename stored in zpaq is a mix of UTF-8 and UTF-16.

[fcorbelli]

void printUTF8(const char* s, FILE* f=stdout)
{
	assert(f);
	assert(s);
	if (g_output_handle!=0)
		fprintf(g_output_handle,"%s",s); <<<---- on log, write just as it is
	if (flagsilent)
		return;
#ifdef unix
  fprintf(f, "%s", s);
#else
  const HANDLE h=(HANDLE)_get_osfhandle(_fileno(f));
  DWORD ft=GetFileType(h);
  if (ft==FILE_TYPE_CHAR) {
    fflush(f);
    std::wstring w=utow(s, '/');
    DWORD n=0;
    WriteConsole(h, w.c_str(), w.size(), &n, 0);
  }
  else
    fprintf(f, "%s", s);
#endif // corresponds to #ifdef (#ifdef unix)
}

[LXY1226]

As I mentioned in my last comment, decompression from Windows to Windows is also fine. However, decompressing mixed names to other platforms may cause issues.