libsmb2 not supporting certain characters

[geofstro]

I opened an issue on the AMSMB2 pages, since it is not allowing to extract data from files with certain latin characters (e.g. ñ, ì).

I notice some work was done here recently on unicode and hope this will fix the problem.

Is it enough to do the following…

git clone https://github.com/amosavian/AMSMB2
cd AMSMB2/buildtools
./build.sh

To update to the latest libsmb2 within AMSMB2? Or does the Author of AMSMB2 need to do something?

Apologies, as this is not really an issue with libsmb2 as such, I believe; just me not understanding how to ensure I'm using the latest version with AMSMB2, and assuming my unicode issue has been fixed in libsmb2 itself with the latest pull request.

[sahlberg]

I think so.
Note, I do not have access to IOS and can not test AMSMB2 but I think that should work.

[geofstro]

Thanks for your response. I rebuilt libsmb2 using the above commands to produce the IOS version. When I tested, it did not cure my problem with those latin characters from Windows shares.

So right now, I don't know if:

a) The rebuilt version includes the fix for this, or if it's just resulting in the same version of libsmb2 for IOS being built, that was already in the AMSMB2 distrib.

b) Whether the unicode changes submitted here, did cure this problem. Perhaps it was just wishful thinking on my part.

I'm referring, of course, to this fix:

#207

[geofstro]

Another thought; can you tell me if libsmb2 correctly handles characters such as ñ, ì, in file names, outside of IOS, MacOS and TVOS? In other words, outside of the AMSMB2 Swift wrapper library? Again, I’m referring to the ability to extract data from files with these characters, download the files, move/rename them. They are listed correctly; but the above features don’t work for me using AMSMB2.

…

On 1 Aug 2022, at 23:25, Ronnie Sahlberg ***@***.***> wrote: I think so. Note, I do not have access to IOS and can not test AMSMB2 but I think that should work. — Reply to this email directly, view it on GitHub <#231 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAVM4KIIMGWR2GGBDH4T62TVXA6GLANCNFSM55G3KO3Q>. You are receiving this because you authored the thread.

[delins]

I don't have time to dive very deep into this at the moment, but my first guess is that it should have worked even before #208. ñ can be expressed using a single UTF-16 unit and therefore the #208 fix wasn't necessary. Unless Swift does some transcoding as is talked about in this so post before handing the filename off to libsmb2, but I doubt that.

How did you rebuild AMSMB2 with libsmb2? AMSMB2 comes bundled with its own statically built version of libsmb2 and refers to it in https://github.com/amosavian/AMSMB2/blob/master/Package.swift. My guess is that you should replace the static libraries in https://github.com/amosavian/AMSMB2/tree/master/libsmb2/lib with the ones you built yourself. If you built libsmb2 as a dynamic library (.so) that won't do you any good in this case since it won't get picked up.

I've been using libsmb2 with the #208 fix for a while now and haven't got any issues with weird unicode characters anymore. But I only use dirlisting at the moment, no file reading, copying or renaming.

Could you name a file abc😀.txt and test how AMSMB2 handles that with your use cases? The smiley uses a surrogate pair in UTF-16, pehaps we learn something from the results.

Also, did you do anything with the locale of your application? My guess is that by default everything in iOS uses UTF-8 (which should be OK), but if you changed it that might cause issues with character encoding.

[geofstro]

I understand and thanks for your response anyway.

I tried naming an example file named abc😀.flac (flac because a music file) and got the same error as with the ñ character…

NSLocalizedFailureReason=Error code 2: Open failed with (0xc0000034) STATUS_OBJECT_NAME_NOT_FOUND.}

As I mentioned before, I have no issue with listing these files either; but I need to extract the data from them, move or download them. Swift uses UTF-8. So I don’t know if that would be an issue. I don’t think so either.

I understand it’s a licensing requirement of AMSMB2 to include libsmb2 as a static library, if I plan to distribute my app commercially, which I intend to do.

From the read me of AMSMB2…

"While source code shipped with project is MIT licensed, but it has static link to libsmb2 which is LGPL v2.1, consequently the whole project becomes LGPL v2.1.

You must link this library dynamically to your app if you intend to distribute your app on App Store."

It looks to me as though libsmb2 is not able to fully support filenames with latin characters from Windows shares. The same problem occurred with the emoji in the filename.

I consider this a significant limitation!

[delins]

I'm not a licensing lawyer so don't take it as fact, but I'm pretty sure it's the opposite: as libsmb2 is lgpl you need to link to it dynamically, ie ship it alongside your application as an .so file so that it's loaded when your application starts, and not link against the .a file. Perhaps this is what you meant.

I agree it would be a significant limitation.

Btw how do you load the file paths into your program? Do you perform a dirlisting with libsmb2, and then use the returned filenames to create the file path, or do you load the paths from a file, stdin, or hardcoded in your application? Any of thoses places may mess up the encoding as well, especially if you develop on windows.

[geofstro]

ie ship it alongside your application as an .so file so that it's loaded when your application starts, and not link against the .a file. Perhaps this is what you meant.

Yes, that is what I meant. Sorry for the confusion.

Do you perform a dirlisting with libsmb2, and then use the returned filenames to create the file path

This is what I do. For Windows, strangely I have to drop the first slash from the returned filename to be able to fetch the data, which works when those characters aren't present. For Mac and Linux, the first slash didn't need to be dropped.

[delins]

I've been reading up on Swift string encodings. It appears that prior to Swift 5, it used either ASCII or UTF-16. I'm not sure which would have been default, but I guess it's ASCII. AMSMB2 creates strings from C's cstring wherever it interfaces with libsmb2, and my best guess is that it somewhere interprets the names as ASCII strings instead of UTF-8 strings (which is what libsmb2 returns).

You might try peppering explicit UTF-8 interpretations into AMSMB2's cString stuff, or perhaps just hardcode a path with the ñ character into some test code to see what that does (and make sure that the Swift string you create with it interprets the string as UTF-8).

[geofstro]

I think I read somewhere Swift used UTF-16 prior to Swift 5. Thanks for the ideas, I will try them out. I don't understand why this is only an issue for Windows SMB shares though. All characters in file names work just fine when the SMB share is MacOSX or Linux.

[delins]

Ah I didn't know you didn't have this problem with MacOSX and Linux shares. That's odd indeed, as far as I know the SMB specification states that path encodings should be UTF-16 LE, and there's no way for a server to specify something else. It may have something to do with the SMB versions, but I doubt it.

[geofstro]

I should have probably made that clear in the title, as I did on the AMSMB2 issues page.

[geofstro]

I have changed the title back again, because further testing has shown that libsmb2 cannot handle these characters on Linux either.

A major problem as most NASs are Linux based.

I will do some further testing on Mac to determine if MacOS is the one exception.