[Raspian][v0.7.0] LookupError: unknown encoding: utf-8

[Fabioamd87]

Hello, I've realized that wsdd is in Raspberry Pi OS bookworm, and this is very cool!
Unfortunately when I start the service trough systemd, I get this error:

Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/sbin/wsdd", line 1165, in handle_new_address
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/sbin/wsdd", line 753, in __init__
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/sbin/wsdd", line 777, in send_hello
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/sbin/wsdd", line 311, in build_message
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/sbin/wsdd", line 427, in xml_to_buffer
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1098, in tostring
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 731, in write
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/lib/python3.11/contextlib.py", line 137, in __enter__
Jan 18 19:23:41 rpi wsdd[55131]:   File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 794, in _get_writer
Jan 18 19:23:41 rpi wsdd[55131]: LookupError: unknown encoding: utf-8

I probably know how to fix this, but is there any solution that doesn't imply to touch the "original" code of the package?

thanks!

[christgau]

Hello, I've realized that wsdd is in Raspberry Pi OS bookworm, and this is very cool!

If you mean wsdd is very cool. Thanks. If not than credits goes to the package maintainer. 😉

Jan 18 19:23:41 rpi wsdd[55131]: LookupError: unknown encoding: utf-8

I doubt that this is an wsdd error. Try to run this in the Python REPL:

print('abc'.encode('utf-8'))

If that does not work, I would say your Python installation is broken or the is system missing iconv (or similar stuff).

[Fabioamd87]

of course wsdd is cool, but what's best that having it just one apt-get install away? :)

this is the output of the command you asked me:

fabio@rpi:~ $ python3
Python 3.11.2 (main, Mar 13 2023, 12:18:29) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> print('abc'.encode('utf-8'))
b'abc'
>>>

i didn't touch my python installation that I remember, it's the default raspberry one.

[Fabioamd87]

I've just notice that the service is run by wsdd user as per service file attached (from debian bookworm)

wsdd.service.txt

but the user is missing on my system:

fabio@rpi:~ $
fabio@rpi:~ $ cat /etc/passwd | grep wsdd
fabio@rpi:~ $

fabio@rpi:~ $
fabio@rpi:~ $ sudo -u wsdd python3 /usr/sbin/wsdd --shortlog --chroot /run/wsdd
sudo: unknown user wsdd
sudo: error initializing audit plugin sudoers_audit
fabio@rpi:~ $

I've noticed this beccause if I manually run wsdd as my user I don't have this issue and I can see my host in windows.

but adding the user wsdd doesn't solve it either:

fabio@rpi:~ $ sudo -u wsdd python3 /usr/sbin/wsdd --shortlog --chroot=/run/wsdd
WARNING: no interface given, using all interfaces
ERROR: could not chroot to /run/wsdd: [Errno 2] No such file or directory: '/run/wsdd'
fabio@rpi:~ $

[christgau]

I've just notice that the service is run by wsdd user as per service file attached (from debian bookworm)
but the user is missing on my system:

It does not matter. The unit file makes use of the DynamicUser=yes feature which creates user dynamically. It's a systemd feature.

I've noticed this beccause if I manually run wsdd I don't have this issue and I can see my host if windows.

Ok, so your system appears to work. That's great.

What could be the reason is the usage of chroot in the unit file. You can try to remove that command line argument in the unit file. Issue a systemctl daemon-reload after you made changes to the unit file and then restart the wsdd service. Note that this reduces security.

fabio@rpi:~ $ sudo -u wsdd python3 /usr/sbin/wsdd --shortlog --chroot=/run/wsdd
WARNING: no interface given, using all interfaces
ERROR: could not chroot to /run/wsdd: [Errno 2] No such file or directory: '/run/wsdd'

Well, that has nothing to do with a missing/present user, doesn't it.

[Fabioamd87]

it seems to work now, thanks :)
I've removed this paramter: --chroot=/run/wsdd
I guess I can accept the risk in my home system.
but as I also work in cybersecurity I would like to fix this issue as well, in case it could help others as well.

[christgau]

Ok. It's very likely that some information stored in a file is missing at runtime due to chroot'ing into an empty directory. Seems to be a fundamental flaw, but needs some further investigation.

[Fabioamd87]

Please let me know if I can help anyhow.

[JedMeister]

FWIW I just installed wsdd on a Debian 12/Bookworm server I have handy (and double checked on my Bookworm desktop too) and it "just works"!?:

root@tkldev8 ~# systemctl status wsdd
* wsdd.service - Web Services Dynamic Discovery host daemon
     Loaded: loaded (/lib/systemd/system/wsdd.service; enabled; preset: enabled)
     Active: active (running) since Fri 2024-01-19 01:09:13 UTC; 13s ago
       Docs: man:wsdd(8)
   Main PID: 2408564 (python3)
      Tasks: 1 (limit: 2312)
     Memory: 18.6M
        CPU: 627ms
     CGroup: /system.slice/wsdd.service
             `-2408564 python3 /usr/sbin/wsdd --shortlog --chroot=/run/wsdd

Jan 19 01:09:13 tkldev8 systemd[1]: Started wsdd.service - Web Services Dynamic Discovery host daemon.
Jan 19 01:09:13 tkldev8 wsdd[2408564]: WARNING: no interface given, using all interfaces

However, I note that the service requires CAP_SYS_CHROOT privilege (so even though it runs as wsdd user, it needs to be started as root to get that capability). See the service file (AmbientCapabilities within the Service section):

root@tkldev8 ~# cat /lib/systemd/system/wsdd.service
[Unit]
Description=Web Services Dynamic Discovery host daemon
Documentation=man:wsdd(8)
; Start after the network has been configured
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
; The service is put into an empty runtime directory chroot,
; i.e. the runtime directory which usually resides under /run
EnvironmentFile=/etc/default/wsdd
ExecStart=/usr/sbin/wsdd --shortlog --chroot=/run/wsdd $WSDD_PARAMS
DynamicUser=yes
User=wsdd
Group=wsdd
RuntimeDirectory=wsdd
AmbientCapabilities=CAP_SYS_CHROOT

[Install]
WantedBy=multi-user.target

I wonder if the Raspberry Pi OS kernel doesn't have that capability? I'd be surprised if it didn't but who knows?

To check if your kernel supports that, try this:

capsh --print | grep -i CAP_SYS_CHROOT

(If you don't have that command, install the libcap2-bin package).

If it doesn't return any output then your kernel doesn't have that capability and I recommend reporting a bug to Raspberry Pi OS.

If your kernel does have that capability, then I suggest checking that the package is coming from Debian (I suspect it is but good to check):

root@tkldev8 ~# apt policy wsdd
wsdd:
  Installed: 2:0.7.0-2.1
  Candidate: 2:0.7.0-2.1
  Version table:
 *** 2:0.7.0-2.1 500
        500 http://deb.debian.org/debian bookworm/main amd64 Packages
        100 /var/lib/dpkg/status

If your output looks the same (or at least very similar - except the amd64), I recommend lodging a bug with Debian - or if it's coming from Raspberry Pi OS, then report it to them. Perhaps it's a Raspberry Pi and/or Raspberry Pi OS and/or ARM specific bug? (As a python script I wouldn't have thought so, but who knows...?)

To report to Debian, install the reportbug package, run the reportbug exectuable (from the CLI) and follow the prompts. I'm not sure about reporting bugs to Raspberry Pi OS?

FWIW I note that there are currently no existing Debian bugs related to wsdd. So if you report there, you'll be the first Debian bug report! 😁

[christgau]

@Fabioamd87 There has been a similar issue quite a while a ago which I am able to reproduce. However, I tried to produce a MWE for this particular issue here:

import os                                                                      
import xml.etree.ElementTree as ET

os.chroot("/var/empty")

xml = ET.Element('item', {'key': u'😉'})
retval = b'<?xml version="1.0" encoding="utf-8">'
print(retval + ET.tostring(xml, encoding='utf-8'))

(let's call that thing the wsdd MWE)

This resembles what is done in the context of the exception reported in the OP.

wsdd/src/wsdd.py

Lines 426 to 427 in c9cffc5

	retval = b'<?xml version="1.0" encoding="utf-8"?>'
	retval = retval + ElementTree.tostring(xml, encoding='utf-8')

However, the MWE works for me with Python 3.9 to 3.12 on Gentoos (sorry, don't have a Raspian installation at hand) and on OpenBSD with 3.9 when chrooting into /var/empty.

What I noticed with the MWE from the Python bugtracker is that indeed the Python interpreter fails to open an encoding-related file between chrooting and bailing out the exception. Here's the strace

chroot("/var/empty")                    = 0
newfstatat(AT_FDCWD, "/usr/lib/python-exec/python3.10/../../../lib/python3.10/encodings", 0x7ffd34cd7410, 0) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/python-exec/python3.10/../../../lib/python3.10/encodings", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
write(2, "Traceback (most recent call last"..., 35) = 35

I can work around the issue in bugtracker MWE if I do an additional urlopen before chrooting which forces a load of the encoding data that appears to be re-used lateron. Nevertheless, I wonder why the exception does not happen with the wsdd MWE for me.

@Fabioamd87 can you give the "wsdd MWE" from a above a try on your RasPi and chroot (you need root privileges for this)

[Fabioamd87]

> 
> I wonder if the Raspberry Pi OS kernel doesn't have that capability? I'd be surprised if it didn't but who knows?
> 
> To check if your kernel supports that, try this:
> 
> ```
> capsh --print | grep -i CAP_SYS_CHROOT

fabio@rpi:~ $ capsh --print | grep -i CAP_SYS_CHROOT
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore
fabio@rpi:~ $

> ```
> 
> (If you don't have that command, install the `libcap2-bin` package).
> 
> If it doesn't return any output then your kernel doesn't have that capability and I recommend reporting a bug to Raspberry Pi OS.
> 
> If your kernel does have that capability, then I suggest checking that the package is coming from Debian (I suspect it is but good to check):
> 
> ```
> root@tkldev8 ~# apt policy wsdd
> wsdd:
>   Installed: 2:0.7.0-2.1
>   Candidate: 2:0.7.0-2.1
>   Version table:
>  *** 2:0.7.0-2.1 500
>         500 http://deb.debian.org/debian bookworm/main amd64 Packages
>         100 /var/lib/dpkg/status
> ```

fabio@rpi:~ $ apt policy wsdd
wsdd:
Installed: 2:0.7.0-2.1
Candidate: 2:0.7.0-2.1
Version table:
*** 2:0.7.0-2.1 500
500 http://deb.debian.org/debian bookworm/main arm64 Packages
500 http://deb.debian.org/debian bookworm/main armhf Packages
100 /var/lib/dpkg/status
fabio@rpi:~ $

fabio@rpi:~ $ wsdd MWE
usage: wsdd [-h] [-i INTERFACE] [-H HOPLIMIT] [-U UUID] [-v] [-d DOMAIN] [-n HOSTNAME] [-w WORKGROUP] [-A] [-t] [-4]
[-6] [-s] [-p] [-c CHROOT] [-u USER] [-D] [-l LISTEN] [-o] [-V]
wsdd: error: unrecognized arguments: MWE
fabio@rpi:~ $

[christgau]

fabio@rpi:~ $ wsdd MWE

@Fabioamd87 Sorry, that's not what I meant. Please do the following:

1. Put that code in a file named mwe.py:

import os                                                                      
import xml.etree.ElementTree as ET

os.chroot("/var/empty")

xml = ET.Element('item', {'key': u'😉'})
retval = b'<?xml version="1.0" encoding="utf-8">'
print(retval + ET.tostring(xml, encoding='utf-8'))

2. run the code with root privileges:

$ sudo python mwe.py

[JedMeister]

@Fabioamd87 - everything looks pretty reasonable there. Your kernel clearly fulfills the requirement and the package is coming from Debian repos (which TBH I'm not surprised about). I wonder if there is some issue with your locale? FWIW I tried to reproduce the issue by removing UTF-8 encoding on a system and even that works for me?! Very weird...

@christgau - FWIW on a Debian based system (unless the python-is-python3 package is installed) it will need to be:

sudo python3 mwe.py

[Teemes]

Hello, this issue has been open for a while now. I found than when I set a non-UTF-8 locale on a new system the LookupError will occur. Changing the locale to a UTF-8 locale and rebooting fixes the issue. Hope that helps.

[7djs7]

This worked for me as well. Updated the locale to UTF-8 and restarted the service. Now working properly!

Hello, this issue has been open for a while now. I found than when I set a non-UTF-8 locale on a new system the LookupError will occur. Changing the locale to a UTF-8 locale and rebooting fixes the issue. Hope that helps.