Add instructions for cross compilation

[mfreeborn]

I think a sizeable use-case for this library is to develop libcamera applications for raspberry pi devices - at least that is my use case. The problem is that these devices can be very underpowered with regards to software development and compiling programs.

I have therefore been spending the last couple of days on and off trying to get cross compilation working - I think I'm getting pretty close now.

I am using this repo to provide the cross compilation toolchain binaries, as well as a basic sysroot directory. It is missing a couple of things:

1. GLIBC is v2.27, but we need at least v2.30. I think this is just a case of updating the configs and re-running the github action on the above-linked repo.

2. It doesn't have an up to date libcamera compiled within it. I have currently circumvented this problem by compiling libcamera on my raspberry pi 0 and copy the appropriate files to the sysroot of the above toolchain.

There is one more oddity that I have come across, regarding stream.h. The static assertion on line 24 static_assert(sizeof(struct libcamera_stream_configuration) == sizeof(libcamera_stream_configuration_t)); fails (112 == 88)... and I'm not sure why (something to do with host being 64-bit and target being 32-bit?) . Naively commenting the line out at least lets me compile camera_c_api, before getting the errors numbered above. I don't know why that static assertion is required in the first place, or why those two sizes would be different.

[chemicstry]

Usually with Rust cross-compilation is quite easy:

• rustup target add aarch64-unknown-linux-gnu
• cargo build --target aarch64-unknown-linux-gnu

bindgen and cc should support cross-compilation out of the box, but the problem is libcamera. I couldn't find any info on cross-compiling except this issue. But for some reason libcamera doesn't want to use different toolchain and still compiles using system gcc. Didn't investigate further.

There is one more oddity that I have come across, regarding stream.h. The static assertion on line 24 static_assert(sizeof(struct libcamera_stream_configuration) == sizeof(libcamera_stream_configuration_t)); fails (112 == 88)... and I'm not sure why (something to do with host being 64-bit and target being 32-bit?) . Naively commenting the line out at least lets me compile camera_c_api, before getting the errors numbered above. I don't know why that static assertion is required in the first place, or why those two sizes would be different.

Yes, this is likely due to 32-bit target. I guess we could add ifdefs to problematic headers. Btw, what raspberry are you using? Both RPi 3 and 4 support 64-bit architecture.

Also, could you explain your workflow? Do you cross-compile and then copy binaries over to raspberry? I only found rust-analyzer with vscode to be slow on RPi 4, but regular incremental builds for libcamera are just a couple of seconds.

[mfreeborn]

Yes, actually I find developing with a remote SSH session in VS Code is a really nice workflow. I'm specifically targeting raspberry pi 0Ws, though which is armv6 32-bit. (`arm-unknown-linux-gnueabihf` on rust).

…

________________________________ From: Jurgis ***@***.***> Sent: Saturday, February 18, 2023 7:34:38 PM To: lit-robotics/libcamera-rs ***@***.***> Cc: Michael Freeborn ***@***.***>; Author ***@***.***> Subject: Re: [lit-robotics/libcamera-rs] Add instructions for cross compilation (Issue #17) Usually with Rust cross-compilation is quite easy: * rustup target add aarch64-unknown-linux-gnu * cargo build --target aarch64-unknown-linux-gnu bindgen and cc should support cross-compilation out of the box, but the problem is libcamera. I couldn't find any info on cross-compiling except this issue<raspberrypi/documentation#1529>. But for some reason libcamera doesn't want to use different toolchain and still compiles using system gcc. Didn't investigate further. There is one more oddity that I have come across, regarding stream.h. The static assertion on line 24 static_assert(sizeof(struct libcamera_stream_configuration) == sizeof(libcamera_stream_configuration_t)); fails (112 == 88)... and I'm not sure why (something to do with host being 64-bit and target being 32-bit?) . Naively commenting the line out at least lets me compile camera_c_api, before getting the errors numbered above. I don't know why that static assertion is required in the first place, or why those two sizes would be different. Yes, this is likely due to 32-bit target. I guess we could add ifdefs to problematic headers. Btw, what raspberry are you using? Both RPi 3 and 4 support 64-bit architecture. Also, could you explain your workflow? Do you cross-compile and then copy binaries over to raspberry? I only found rust-analyzer with vscode to be slow on RPi 4, but regular incremental builds for libcamera are just a couple of seconds. — Reply to this email directly, view it on GitHub<#17 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHSVKWGWJQENA3VQM35U5C3WYEP45ANCNFSM6AAAAAAVAO5RWY>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[kbingham]

libcamera uses meson as it's build system. If you want to cross compile libcamera to aarch64 you have to provide a suitable cross file and environment for that.

Meson documents this at:

• https://mesonbuild.com/Cross-compilation.html

[chemicstry]

Thanks for the tip, I actually succeeded in cross-compiling libcamera, but didn't have time to update. Here are the steps on Ubuntu 20.04 if anyone is interested:

• Install dependencies

  sudo apt install gcc-aarch64-linux-gnu g++-aarch64-linux-gnu pkg-config-aarch64-linux-gnu

• Create aarch64_cross.txt:

  [binaries]
  c = '/usr/bin/aarch64-linux-gnu-gcc'
  cpp = '/usr/bin/aarch64-linux-gnu-g++'
  ar = '/usr/bin/aarch64-linux-gnu-gcc-ar'
  strip = '/usr/bin/aarch64-linux-gnu-strip'
  pkgconfig = '/usr/bin/aarch64-linux-gnu-pkg-config'
  
  [host_machine]
  system = 'linux'
  cpu_family = 'aarch64'
  cpu = 'aarch64'
  endian = 'little'
  

• Build:

  meson cross-build --cross-file aarch64_cross.txt
  ninja -C cross-build

I got stuck on pkg-config and sysroot stuff though, because I have no idea how they work. Doing ninja -C cross-build install installs aarch64 binaries into system folders instead of under arch-specific subfolder. Some guides on the internet suggested setting up multi-arch system via apt, but I'm not sure if that is neccessary. The rust pkg-config lib gives the following error (hint):

Install a sysroot for the target platform and configure it via PKG_CONFIG_SYSROOT_DIR and PKG_CONFIG_PATH, or install a cross-compiling wrapper for pkg-config and set it via PKG_CONFIG environment variable.

I will update if I get any further.

[kbingham]

Remember if you're cross compiling like that - then it will only 'work' if you're cross compiling on ubuntu-20.04 on your 'host', and your target device is also running ubuntu-20.04. You can't expect to cross compile on ubuntu-20.04 (x86) and target raspberry-pi-bullseye like that for instance. There will be all sorts of conflicts with different library versions and other dragons.

You can however have a bullseye-arm64 root filesystem set as the sysroot to compile against if it matches what the target will have. That's what sysroots are for.

I had an intern tackle this with docker to handle the requirements in the past which might have something helpful/useful:

https://github.com/CactiChameleon9/Libcamera-RPiOS-Build-Enviroment

[mfreeborn]

I've spent too long on this - I managed to get it working transiently, but I think the cost-benefit isn't there.

Cross compiling either rust or libcamera is doable enough, but I've run into endless brick walls trying to get it to all work together in a sufficiently simple manner.

At one stage I ended up juggling the Ubuntu build system's filesystem, a debootstrap filesystem, a sysroot from this cross compiler set up and finally the filesystem on the target machine. Throw in pkg-config and multiarch issues, dynamic linking of standard libraries from all over the place, the fact you are trying to compose meson cross compilation (and cross compilation of optional extras, rust cross compilation, building the C wrapper in rust (with cc) and generating the bindings to the C wrapper in rust and throw in the fact that debian armhf != Raspberry Pi armhf... I could go on.

So my conclusion is that whilst it's possible, it's not practical.

If one is targeting a higher end Raspberry Pi, just use something like VS Codes SSH remote development. If one is targeting e.g. a Raspberry Pi 0 (like me), then just develop the application on your development machine (I need to research how to use the vimc driver, or use a cheapo usb camera) and make a coffee whilst you do your one final native compilation on the Pi.

[kbingham]

for vimc ... just 'sudo modprobe vimc'.

[kbingham]

If you have energy left, It could still be worth retrying what you've done with the docker image from https://github.com/CactiChameleon9/Libcamera-RPiOS-Build-Enviroment and extending that instead of https://github.com/tttapa/docker-arm-cross-toolchain as the cross compiler we use is installed directly from the bullseye repo, where as the 'docker-arm-cross-toolchain' is custom creating something.

[mfreeborn]

Thanks for the tip about vimc.

I believe that one of the issues that tttapa's setup solves (at least for 32-bit systems) is that debian armhf supports armv7 CPUs, whilst Raspberry Pi's opinion of armhf is armv6. Therefore, the default debian armhf builds are not necessarily compatible (I suspect they might work for a bit... until they don't).

[kbingham]

Ah, wait - are you trying to use the rpi-zero *original? Or the RPi zero 2 ?

[mfreeborn]

The original, although I have stock alerts set up for the zero 2, which would make things simpler. At least, compiling on it would be much quicker, although it doesn't have enough RAM to do remote development on it practically.

…

On Tue, 21 Feb 2023 at 16:34, Kieran Bingham ***@***.***> wrote: Ah, wait - are you trying to use the rpi-zero *original? Or the RPi zero 2 ? — Reply to this email directly, view it on GitHub <#17 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHSVKWBLN3V4TXM2GVLXJ3DWYTVCLANCNFSM6AAAAAAVAO5RWY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[fishrockz]

Hi

I have been doing some embedded projects and have cross compiled one of your examples in my experimental integration tool https://gitlab.com/pointswaves/girderstream-test-project/-/blob/264bd9a4ea87da9af8cd9ccdc724e153087ae85d/elements/libcamera-rs.grd (the only reason i have not build more of your examples is because my cross tool chain does not have a update enough rust for your other ones. Updating this is on my todo list lol)

I use yocto/buildstream/buildroot at work but i have done the above with my experimental project girderstream. I only mention this so i can show you my example while also noting that its not the common way but that i do do quite a lot of embedded.

My way of cross building libcamera-rs, in plain english, so you don't need to go look at my example in a unusual build file is as follows.

Place your previously cross-compiled libcamera and its dependencies in a directory, in this case /targetfs/

You will then need to configure pkgconfig to find your cross compiled libcamera, ie were is the libcamera.pc file etc

export PKG_CONFIG_PATH=/targetfs/usr/lib/pkgconfig/
export PKG_CONFIG_LIBDIR=/targetfs/usr/lib/pkgconfig/
export PKG_CONFIG_SYSROOT_DIR=/targetfs/

At this point pkgconfig can find everything so you should just need to tell cargo that you want to cross compile

First update your .cargo/config so it knows which linker to use etc

[target.aarch64-unknown-linux-gnu]
linker = "aarch64-unknown-linux-gnu-gcc"
rustflags = ["-Clink-args=-Xlinker -rpath-link=/targetfs/usr/lib/%{target-triplet}/"]

Tell cargo/rustc how to build the C wrapper in libcamera-rs-sys by telling it what c++ compiler to use.
export TARGET_CXX=your-cross-compiler-g++"

Then compile the example
cargo build --release --example version --target=aarch64-unknown-linux-gnu

The above is all pretty standard for cross building a libABC-rs-sys except for having to set the rpath

I have had to manually set the rpath, so the linker can find the .so files of the dependnecies in the /targetfs/ which is not on the default linker path

As you can see the version example links to many of the libcamera dependencies

/ # /lib/ld-linux-aarch64.so.1 --list /version
	linux-vdso.so.1 (0x0000ffffada53000)
	libcamera.so.0.0.5 => /usr/lib/aarch64-linuxgrd-gnu/libcamera.so.0.0.5 (0x0000ffffad7e0000)
	libstdc++.so.6 => /usr/lib/aarch64-linuxgrd-gnu/libstdc++.so.6 (0x0000ffffad5b0000)
	libgcc_s.so.1 => /usr/lib/aarch64-linuxgrd-gnu/libgcc_s.so.1 (0x0000ffffad580000)
	libc.so.6 => /usr/lib/aarch64-linux-gnu/libc.so.6 (0x0000ffffad3d0000)
	/lib/ld-linux-aarch64.so.1 (0x0000ffffada1b000)
	libcamera-base.so.0.0.5 => /usr/lib/aarch64-linuxgrd-gnu/libcamera-base.so.0.0.5 (0x0000ffffad390000)
	libgnutls.so.30 => /usr/lib/aarch64-linuxgrd-gnu/libgnutls.so.30 (0x0000ffffad140000)
	libyaml-0.so.2 => /usr/lib/aarch64-linuxgrd-gnu/libyaml-0.so.2 (0x0000ffffad110000)
	libm.so.6 => /usr/lib/aarch64-linux-gnu/libm.so.6 (0x0000ffffad070000)
	libunwind.so.8 => /usr/lib/aarch64-linuxgrd-gnu/libunwind.so.8 (0x0000ffffad030000)
	libnettle.so.8 => /usr/lib/aarch64-linuxgrd-gnu/libnettle.so.8 (0x0000ffffacfd0000)
	libhogweed.so.6 => /usr/lib/aarch64-linuxgrd-gnu/libhogweed.so.6 (0x0000ffffacf60000)
	libgmp.so.10 => /usr/lib/aarch64-linuxgrd-gnu/libgmp.so.10 (0x0000ffffaced0000)
/ # ./version 
libcamera: v0.0.5

The way i think this should work is that you should provide both the -L (linker) and the -I (include) paths instead of just the -I .clang_arg(format!("-I{}", libcamera_include_path.display()))

The docs for pkgconfig https://docs.rs/pkg-config/latest/pkg_config/struct.Library.html talk about the -I and -L paths.

I think the reason you don't normally need to set the link path is because up until now all the required .so files were on the default link paths, but if you ever tried to build libcamera-rs-sys on a system were the .so of the deps were not on the default link path, i think you would have this issue.

So would you like me to write some patches to:

• Pass on the link path between pkgconfig and the libcamera-rs-sys bindgen builder?
• Write up the above in to the readme?

[chemicstry]

@fishrockz wow, thanks for the in depth explanation.

Pass on the link path between pkgconfig and the libcamera-rs-sys bindgen builder?

Sure

Write up the above in to the readme?

Maybe you could write this as a separate file (i.e. CROSSCOMPILING.md) and link it from the main README? Also, could you include how to cross-compile libcamera itself? Unless there is a howto somwhere else that we can link to.

[shuanglengyunji]

I fighted with the cross toolchain for a couple of days and get cross compilation work on both x86_64 and aarch64 targets.
I may find time to add CROSSCOMPILING.md, before that I hope the following docker configurations will help.

The following 2 docker configurations work with cross build --target aarch64-unknown-linux-gnu and cross build --target x86_64-unknown-linux-gnu.

To configure cross:

1. My Cross.toml

[build.env]
passthrough = ["RUST_DEBUG"]

[target.x86_64-unknown-linux-gnu]
dockerfile.file = "docker/Dockerfile.x86_64"
dockerfile.context = "docker"

[target.aarch64-unknown-linux-gnu]
dockerfile.file = "docker/Dockerfile.aarch64"
dockerfile.context = "docker"

For aarch64 (Raspberry Pi 5) running bookworm based Raspberry Pi OS:

1. I installed libcamera-dev to raspberry pi os, and copy off the rootfs into a tarball (I assme you can also cross compile the libcamera from raspberry pi official fork)
2. My dockerfile:

FROM ghcr.io/cross-rs/aarch64-unknown-linux-gnu:main

# Unpack the rpi_os_lite_bookworm_rootfs.tar.gz file into /target_fs/
# Note this rootfs needs to be equal or order than Debian 12 / Raspberry Pi OS Bookworm
COPY rpi_os_lite_bookworm_rootfs.tar.gz /target_fs/
RUN tar -xzf /target_fs/rpi_os_lite_bookworm_rootfs.tar.gz -C /target_fs/
RUN rm /target_fs/rpi_os_lite_bookworm_rootfs.tar.gz

# Install GCC-12 to match the target raspberry pi os bookworm (Debian 12)
RUN apt-get update
RUN apt-get install -y wget gnupg && echo "deb http://deb.debian.org/debian bookworm main" > /etc/apt/sources.list.d/debian.list
RUN apt-get update
RUN apt-get install -y \
    # Libcamera dependencies
    gcc-12-aarch64-linux-gnu \
    g++-12-aarch64-linux-gnu \

# set env variables for cross-compilation
ENV SYSROOT=/target_fs
ENV PKG_CONFIG_SYSROOT_DIR=$SYSROOT
ENV PKG_CONFIG_PATH=$SYSROOT/usr/lib/aarch64-linux-gnu/pkgconfig
ENV PKG_CONFIG_LIBDIR=$SYSROOT/usr/lib/aarch64-linux-gnu/pkgconfig
ENV CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc-12
ENV RUSTFLAGS="-C link-args=--sysroot=$SYSROOT -C target-cpu=cortex-a76"

# Set up the sysroot for QEMU emulation runtime
ENV QEMU_LD_PREFIX=$SYSROOT

# Bindgen uses clang to generate bindings, so we need to set the sysroot and include paths
ENV BINDGEN_EXTRA_CLANG_ARGS_aarch64_unknown_linux_gnu="--sysroot=$SYSROOT -I$SYSROOT/usr/include/aarch64-linux-gnu/c++/12"

For x86_64

1. My Dockerfile:

FROM ghcr.io/cross-rs/x86_64-unknown-linux-gnu:main

# Update the package list
RUN apt-get update

# install libyaml-dev python3-yaml python3-ply python3-jinja2 python3-pip libssl-dev openssl ninja-build pkg-config
RUN apt-get install -y \
	wget \
    gnupg \
	# libcamera dependencies
    libyaml-dev \
    python3-yaml \
    python3-ply \
    python3-jinja2 \
    python3-pip \
    libssl-dev \
    openssl \
    ninja-build \
    pkg-config \

# Install libcamera - Libcamera supports only dynamic linking
RUN pip3 install meson==0.63.0
RUN git clone --depth 1 --branch v0.5.1 https://git.libcamera.org/libcamera/libcamera.git
RUN cd libcamera && meson setup build && ninja -C build install