UTF-8 handling is not functioning with Windows PowerShell

[efc-mw]

With pandoc 3.6.4 I can't get german umlaute and for example the copyright symbol working. The input format is docbook encoded in UTF-8, the output fromat is aciidoc. I tried also html as output format, the same problem

I use pandoc 3.6.4 with windows 11:

The docbook file to reproduce:

<chapter xml:id="USER_INTRO" xmlns="http://docbook.org/ns/docbook"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns:xlink="http://www.w3.org/1999/xlink"
         xmlns:xi="http://www.w3.org/2001/XInclude"
         xmlns:svg="http://www.w3.org/2000/svg"
         xmlns:ns="http://docbook.org/ns/docbook"
         xmlns:mml="http://www.w3.org/1998/Math/MathML"
         xmlns:html="http://www.w3.org/1999/xhtml">
  <title>Einführung®</title>
</chapter>```

[jgm]

This works fine for me on macos.

I suspect you don't have something set up right for UTF-8. Are you sending output to a file or to the console? If to the console, are you sure you have the encoding set correctly there?

In any case, I think it's very unlikely that this is a bug. We've supported Windows for at least 15 years, and if pandoc were not capable of handling UTF-8 on Windows, we would know it. No recent changes would have affected this. I'll convert this to a discussion and hopefully another Windows/pandoc user can give advice.

[efc-mw]

THX for your fast reply!

I write to a file. My command is:
pandoc -wrap=none -f docbook -t html test.xml >test.html

After your comment, I re-checked, the funny thing is: I get it working with CMD, but with PowerShell as shell, I got the problem. So I think the bug/configuration problem is with PowerShell/Windows Terminal and not with CMD as shell. For me it's ok, I can use CMD for the conversion. THX

[jgm]

Try with -o test.html instead of console redirection.
That eliminates one source of problems. (What kysko said!)

[kysko]

No problem here, with pandoc 3.6.4 on Win10Home, locale en-CA.
Both

pandoc -wrap=none -f docbook -t html test.xml >test.html

and

pandoc -wrap=none -f docbook -t html test.xml -o test.html

give the same (bitwise) result in powershell (7.5.0) or cmd with your test example (minus the trailing "```").

Try with the -o option on pwsh, to be sure.

/Edit:
Ah! I tried with the old "Windows Powershell", which is version 5.x, and indeed your invocation (without the -o option) wrote the standard output in UTF-16 LE, which then directed to a file with the same encoding.
Can you confirm you use this old powershell?

However, using the -o option properly wrote in UTF-8 from the get go.

So either use the -o option (which is better anyway), or install newer powershell 7.x+! Both is better.

(Installing powershell 7.x+ won't uninstall the old powershell. In batch files, powershell invokes the old version, and pwsh invokes the newer.)

/Edit:
Just a last edit (I hope!): you can in fact change the stdout encoding in the old powershell to utf8 (with bom).
See Character encoding in Windows PowerShell.