DiscImageCreator -> Redumper

[tjanas]

I see there is support for DiscImageCreator in the RetroNAS Install Menu. Since DiscImageCreator is in the midst of being deprecated in favor of redumper for dumping optical media to Redump standards, would you consider adding redumper?

[sairuk]

we'd need to take a look, there a reference from the redump forums or anything you can link too?

[tjanas]

http://wiki.redump.org/index.php?title=Redumper_status

http://forum.redump.org/topic/47761/updated-guidance-for-audio-cd-submissions/

http://wiki.redump.org/index.php?title=Redumper

http://forum.redump.org/topic/45004/audio-offset/

[sairuk]

cool, ta. I took a look at building redumper last night but the tooling in bullseye is behind what redumper needs to build we'd probably have to put this off until rpios moves to bookworm.

Still considering, i'll add it to the ideas page because its a good move. thanks

[tjanas]

http://forum.redump.org/topic/50811/redumper-submission-updates/

[sairuk]

have played with this some more fixing various things to the point it seems to be compiling but never finishes on x86, doesn't build on arm yet.

dropping these rough scripts here for future reference

patch CMakeLists.txt

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 52d112e..a51e199 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -65,6 +65,10 @@ if(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
        # -fexperimental-library is required for clang-16 as std::format is defined there
        add_compile_options(-stdlib=libc++ -fexperimental-library)
        add_link_options(-stdlib=libc++ ${REDUMPER_CLANG_LINK_OPTIONS} -fexperimental-library)
+       add_compile_options(-pthread)
+       add_link_options(-lpthread)
+
+
 
        # silence implicit modules use warning
        # TL;DR, this will be taken cared of by listing module dependencies explicitly (cmake)

redumper_deps

#!/bin/bash

source /etc/os-release
PKGARCH=$(dpkg --print-architecture)
ARCH=$(uname -m)

wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
echo "deb http://apt.llvm.org/${VERSION_CODENAME}/ llvm-toolchain-${VERSION_CODENAME}-17 main" > /etc/apt/sources.list.d/llvm-${VERSION_CODENAME}.list
apt-get update
apt-get remove -y clang-14 lldb-14 lld-14 clang-tools-14 libc++-14-dev libc++abi-14-dev libc++1-14 libc++abi1-14 libunwind-14 libunwind-14-dev python3-lldb-14
apt-get install -y clang-17 lldb-17 lld-17 clang-tools-17 libc++-17-dev libc++abi-17-dev
ln -s /usr/lib/llvm-17/lib/clang/17 /usr/lib/llvm-17/lib/clang/17.0.0
ln -s /usr/lib/llvm-17/lib/libc++experimental.a /usr/lib/$(ARCH)-linux-gnu

apt-get remove -y ninja-build cmake
apt-get install python3-pip

pip install cmake --upgrade --break-system-packages
pip install ninja --upgrade --break-system-packages

ln -s /usr/local/bin/ninja /bin/ninja

redumper_build

#!/bin/bash

export CC=/usr/bin/clang-17
export CXX=/usr/bin/clang++-17

BASE=/opt/redumper
BUILD_DIR=${BASE}
BUILD_TYPE=Release

cd /opt/redumper
mkdir ${BUILD_DIR}

#config
cmake -B ${BUILD_DIR} -G "Ninja" -DCMAKE_BUILD_WITH_INSTALL_RPATH=ON -DCMAKE_BUILD_TYPE=${BUILD_TYPE} -DREDUMPER_CLANG_LINK_OPTIONS="-static"

#build
cmake --build ${BUILD_DIR} --config ${BUILD_TYPE}

[superg]

Got it, thanks for the reply.
I'll try to find some time to see/fix if it builds on Debian 11 x86_64.

By the way, my Raspbian native build failed as system was constantly low on memory. I had to do single threaded build that took half a day but there was some other issue that I don't remember, will have to revisit at some point but I wish there were good prebuilt clang-16 binaries for rpi as I had to build the toolchain before and it took another 2 days...

[sairuk]

thanks for the support appreciate it.

I am going to try increasing the resources on the VM today and retry, see if I get any further. It is my test host for RetroNAS changes so its pretty closely specced (as possible) to an RPI3/4 and i believe (but not enough coffee this morning) it failed with a memory error.

The rpi build barely started for me, failing to link on generate_<something> I cant recall which step but i'll grab the error and update the thread, although I expect eventually it'll fail out as per the the x86_64 vm if i get past these teething problems.

[sairuk]

increasing VM resources (to 8vCPU/16GB ram) resulted in a successful build of the x86_64

output for reference including warnings

root@testronas:/opt/redumper# ~/redumper_build 
mkdir: cannot create directory ‘/opt/redumper’: File exists
CMake Warning (dev) at CMakeLists.txt:114 (target_sources):
  CMake's C++ module support is experimental.  It is meant only for
  experimentation and feedback to CMake developers.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done (0.3s)
-- Generating done (0.2s)
-- Build files have been written to: /opt/redumper
[1/178] Building C object lzma/CMakeFiles/lzma.dir/Alloc.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[2/178] Building C object lzma/CMakeFiles/lzma.dir/Threads.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[3/178] Building C object lzma/CMakeFiles/lzma.dir/LzFindMt.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[4/178] Building C object lzma/CMakeFiles/lzma.dir/LzFindOpt.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[5/178] Building C object lzma/CMakeFiles/lzma.dir/CpuArch.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[7/178] Building C object lzma/CMakeFiles/lzma.dir/LzmaDec.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[20/178] Building C object lzma/CMakeFiles/lzma.dir/LzmaEnc.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[23/178] Building C object lzma/CMakeFiles/lzma.dir/LzFind.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[27/178] Generating driveoffsets.inc
offset min: -1164, offset max: +1776
[29/178] Building C object CMakeFiles/redumper.dir/lzma/CpuArch.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[30/178] Building C object CMakeFiles/redumper.dir/lzma/Alloc.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[31/178] Building C object CMakeFiles/redumper.dir/lzma/LzFindOpt.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[32/178] Building C object CMakeFiles/redumper.dir/lzma/Threads.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[37/178] Building C object CMakeFiles/redumper.dir/lzma/LzFindMt.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[58/178] Building C object CMakeFiles/redumper.dir/lzma/LzmaDec.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[79/178] Building C object CMakeFiles/redumper.dir/lzma/LzmaEnc.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[84/178] Building C object CMakeFiles/redumper.dir/lzma/LzFind.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[130/178] Building CXX object CMakeFiles/redumper.dir/version.ixx.o
warning: module uses __DATE__ or __TIME__ [-Wpch-date-time]
1 warning generated.
[168/178] Building CXX object CMakeFiles/redumper.dir/skeleton.ixx.o
In file included from /opt/redumper/skeleton.ixx:3:
In file included from /usr/lib/llvm-17/bin/../include/c++/v1/algorithm:1778:
In file included from /usr/lib/llvm-17/bin/../include/c++/v1/__algorithm/inplace_merge.h:28:
/usr/lib/llvm-17/bin/../include/c++/v1/__memory/unique_ptr.h:68:5: warning: delete called on 'gpsxre::SectorReader' that is abstract but has non-virtual destructor [-Wdelete-abstract-non-virtual-dtor]
   68 |     delete __ptr;
      |     ^
/usr/lib/llvm-17/bin/../include/c++/v1/__memory/unique_ptr.h:300:7: note: in instantiation of member function 'std::default_delete<gpsxre::SectorReader>::operator()' requested here
  300 |       __ptr_.second()(__tmp);
      |       ^
/usr/lib/llvm-17/bin/../include/c++/v1/__memory/unique_ptr.h:266:75: note: in instantiation of member function 'std::unique_ptr<gpsxre::SectorReader>::reset' requested here
  266 |   _LIBCPP_INLINE_VISIBILITY _LIBCPP_CONSTEXPR_SINCE_CXX23 ~unique_ptr() { reset(); }
      |                                                                           ^
/opt/redumper/skeleton.ixx:196:32: note: in instantiation of member function 'std::unique_ptr<gpsxre::SectorReader>::~unique_ptr' requested here
  196 |         std::unique_ptr<SectorReader> sector_reader;
      |                                       ^
1 warning generated.
[178/178] Linking CXX executable redumper

I don't think we can guarantee a build on device (as i'd intended) for redumper atm since RetroNAS targets rpios and older laptops which may struggle for resources but it is good to know i can get it to build successfully. I'll have to make a note about resources in the doco to this end.

my other option was to pull the github artifacts but unfortunately afaik they are stuck behind auth

[superg]

I fixed all these warnings yesterday ;)

My GitHub actions automatically builds packages and publishes a release, there shouldn't be auth issue if you directly download it from:
https://github.com/superg/redumper/releases/download/build_268/redumper-2023.11.27_build268-Linux.zip

[sairuk]

hrrm thought i checked there yesterday, might be getting confused, thanks for pointing it out.

although i normally like to have a single flow for all platforms that is the easiest solution for x86_64 and we have precident with other tools in this project.

[superg]

The rpi build barely started for me, failing to link on generate_<something> I cant recall which step but i'll grab the error and update the thread, although I expect eventually it'll fail out as per the the x86_64 vm if i get past these teething problems.

Yeah it's generate_offsets helper util that embeds known drive offsets text file. I think the main redumper executable will fail too with the same linker messages. It's usually std::filesystem that requires some additional linking or modules related things. This is my bleeding edge project so yeah, latest tools are needed. I would expect tools to improve in a year so updated compiler/cmake/ninja will be available in main distro repositories.

[sairuk]

rpi4 4GB (deb12) with pthread dep removed, this sat for about 3h before it finally killed the rpi requiring a reboot

had to symlink libc++-experimental.a to get the compile to move forward however
ln -s /usr/lib/llvm-17/lib/libc++experimental.a /usr/lib/gcc/aarch64-linux-gnu/12/

mkdir: cannot create directory ‘/opt/redumper’: File exists
CMake Warning (dev) at CMakeLists.txt:118 (target_sources):
  CMake's C++ module support is experimental.  It is meant only for
  experimentation and feedback to CMake developers.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done (0.2s)
-- Generating done (0.1s)
-- Build files have been written to: /opt/redumper
[3/145] Generating driveoffsets.inc
offset min: -1164, offset max: +1776
[5/145] Building C object CMakeFiles/redumper.dir/lzma/CpuArch.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[6/145] Building C object CMakeFiles/redumper.dir/lzma/Alloc.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[7/145] Building C object CMakeFiles/redumper.dir/lzma/LzFindOpt.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[10/145] Building C object CMakeFiles/redumper.dir/lzma/Threads.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[14/145] Building C object CMakeFiles/redumper.dir/lzma/LzFindMt.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[26/145] Building C object CMakeFiles/redumper.dir/lzma/LzmaDec.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[38/145] Building C object CMakeFiles/redumper.dir/lzma/LzFind.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[61/145] Building C object CMakeFiles/redumper.dir/lzma/LzmaEnc.c.o
clang-17: warning: argument unused during compilation: '-stdlib=libc++' [-Wunused-command-line-argument]
[100/145] Building CXX object CMakeFiles/redumper.dir/version.ixx.o
warning: module uses __DATE__ or __TIME__ [-Wpch-date-time]
1 warning generated.
[129/145] Building CXX object CMakeFiles/redumper.dir/analyzers/silence.ixx.oTimeout, server 192.168.0.136 not responding.

[sairuk]

built aarch64 successfully on an orangepi5 (8/16 config again) had to symlink libc++experimental.a as well

ln -s /usr/lib/llvm-17/lib/libc++experimental.a /usr/lib/gcc/aarch64-linux-gnu/11/

build is here: https://mameau.com/notes/retronas/redumper-2023.11.28-Linux-aarch64.zip

root@orangepi5:/opt/redumper# time ~/redumper_build 
mkdir: cannot create directory ‘/opt/redumper’: File exists
-- The C compiler identification is Clang 17.0.6
-- The CXX compiler identification is Clang 17.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang-17 - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++-17 - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
CMake Warning (dev) at CMakeLists.txt:114 (target_sources):
  CMake's C++ module support is experimental.  It is meant only for
  experimentation and feedback to CMake developers.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done (3.3s)
-- Generating done (0.0s)
-- Build files have been written to: /opt/redumper
[27/174] Generating driveoffsets.inc
offset min: -1164, offset max: +1776
[119/174] Building CXX object CMakeFiles/redumper.dir/version.ixx.o
warning: module uses __DATE__ or __TIME__ [-Wpch-date-time]
1 warning generated.
[174/174] Linking CXX executable redumper
Test project /opt/redumper
    Start 1: tests
1/1 Test #1: tests ............................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.01 sec
CPack: Create package using ZIP
CPack: Install projects
CPack: - Install project: redumper []
CPack: Create package
CPack: - package: /opt/redumper/redumper-2023.11.28-Linux.zip generated.

real	6m58.390s
user	17m2.217s
sys	1m2.081s
root@orangepi5:/opt/redumper# 

root@orangepi5:/opt/redumper# file ./redumper
./redumper: ELF 64-bit LSB executable, ARM aarch64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=05b8ee4c7e2b9868f6af78c552f4512de1b9e532, for GNU/Linux 3.7.0, not stripped
root@orangepi5:/opt/redumper# 

root@orangepi5:/opt/redumper# cat /etc/os-release 
PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
root@orangepi5:/opt/redumper#

[sairuk]

redumper support is now in, I am hosting the aarch64 build until it becomes available officially

[superg]

This is awesome news!
One question, where did you get recent llvm-17 build for rpi, did you build it or there is some prebuilt repository that I can use for mine?

[sairuk]

running it on the bookworm image with the packages from the official llvm repo

build source (opi5)
deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-17 main

rpi bookworm
deb http://apt.llvm.org/bookworm/ llvm-toolchain-bookworm-17 main

[superg]

Perfect, will try that here, thank you!