International (non-ASCII) characters in filenames - Mac can't read

[claud9999]

How do I get Netatalk to support International characters, such as Déjà Vu.m4a? It shows up on the Mac in the Finder but the Finder is unable to read it. From the shell it reports "file not found" even though auto-complete completes the filename. My Mac is running Monterey (12.7.6) and I'm trying with the latest Docker hub image.

[claud9999]

Ok here's another test. Déjà Vu-1.m4a is the file that can be read on Ubuntu but not on Mac. I scp'd it to the Mac then moved it to Déjà Vu-2.m4a and it can be read both on Mac and Ubuntu. Here's the hexdump of the two filenames on my Ubuntu system:

cdknight@raid:/raid$ ls -1 D*-1.m4a | hexdump -C
00000000  44 65 cc 81 6a 61 cc 80  20 56 75 2d 31 2e 6d 34  |De..ja.. Vu-1.m4|
00000010  61 0a                                             |a.|
00000012
cdknight@raid:/raid$ ls -1 D*-2.m4a | hexdump -C
00000000  44 c3 a9 6a c3 a0 20 56  75 2d 32 2e 6d 34 61 0a  |D..j.. Vu-2.m4a.|
00000010

[claud9999]

(For now I'm rsync-ing the folder from my Ubuntu system to my Mac and then copying back to shared fs to fix this. All non-ascii characters have this issue.)

[rdmark]

This is an odd one. I have tried to reproduce in my environment, but it works from both ends – when creating the file on the Mac and move to the AFP volume, and when creating the file from within the running Docker container's volume (e.g. touch Déjà\ Vu.m4a) on an Ubuntu 22.04 host. I'm using the macOS 15.1.1 AFP client and the latest 4.0.7 Docker image.

We had a similar-looking issue not long ago in #1667 which turned out to be a build system mishap, that has since been firmly fixed.

Would you be able to turn on debug logging and share what it says when you create and copy the affected files?

[APCCV]

Not a solution but I remembering reading something about this behaviour somewhere.
Here: https://eclecticlight.co/2017/04/06/apfs-is-currently-unusable-with-most-non-english-languages/
And follow up: https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/

So "copying back to shared fs to fix" might be the best workaround available as then MacOS is fixing it the way it should be.

rsync has a --iconv option to convert between UTF-8 NFC & NFD, which may allow you to avoid the extra hop, but that defeats the AFPS purpose.

I'm playing with iconv and netatalk as I type this... maybe if you use a netatalk build -Dwith-libiconv=true some of the issues will be addressed. Not sure how to get a docker image to use libiconv or if it already does.

[JDLH]

This seems to me a case of that problem where Netatalk expect filenames on the server's file system to have certain relationships with the filename seen by macOS clients, and when you use scp to put a file onto the server's file system, you are giving it a filename which does not have those relationships.

Howard Oakley's **Explainer: Unicode, normalization and APFS ** (the second link contributed by @APCCV) does indeed describe the issue. It is Unicode normalisation.

See your "Déjà Vu.m4a" examples above.

• In "Déjà Vu-1.m4a", the "é" is represented by bytes 65 cc 81. These are the two Unicode characters U+0065 SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT, in UTF-8 encoding. The "à" is represented by bytes 61 cc 80. These are the two Unicode characters U+0061 SMALL LETTER a followed by U+0300 COMBINING GRAVE ACCENT.
• In "Déjà Vu-2.m4a", the "é" is represented by bytes c3 a9. These are the one Unicode character U+‎00E9 LATIN SMALL LETTER E WITH ACUTE, in UTF-8 encoding. The "à" is represented by bytes c3 a0. These are the one Unicode character 00E0 LATIN SMALL LETTER A WITH GRAVE.

These two representations mean the "same thing", but have different byte sequences. Unicode normalisation is the process of representing all character sequences which mean the "same thing" by the same byte sequences. There are multiple kinds of normalisation. Linux chose one. MacOS chose another. Netatalk requires that the filename on the server's file system uses the normalisation form (the exact byte sequence) which macOS expects. You could imagine a world where Netatalk would accept filenames in either form, and smooth over the difference, but that is apparently not the world we live in.

Another example of the same issue: server file systems which are case sensitive. I had a situation once where I had two files, 'Example' and 'example', on the server file system. Netatalk presented them to the mac client as two files named the same thing (I forget which letter case it used). That is not supposed to happen on macOS! Needless to say, this was very confusing.

One simple way to deal with this issue is to avoid storing any files on the server filesystem under Netatalk except via Netatalk. Another simple (but not easy!) way to deal with it is to avoid using filenames which have different Unicode normalisation forms. Unaccented letters, and Chinese characters, are examples of these. But it is not easy to know which filenames include characters that have different Unicode normalisation forms. A more difficult way to deal with it is to have whatever software stores files to the filesystem under Netatalk take responsibility for normalising filenames the way that Netatalk will expect for presentation to macOS clients.