Skip to content

Commit 8107160

Browse files
brainstormbrainstorm
authored
Musl support for htslib 1.10.2 (#193)
* Add alternative strategy (local dependencies install) to get musl support shipped into rust-htslib * Tweaking toolchain further for rust-htslib-musl and its dependencies * Circumvent Makefile openssl bug in openssl/openssl#11362 and many other compile time gotchas * Add newly docker baked image with the whole toolchain built statically with and for musl * Bypass unneeded CI steps for now to focus on musl * Tell openssl-sys crate where is our custom musl-openssl cross compiled directory... * It is possible that https://github.com/rust-bio/rust-htslib/pull/193/checks?check_run_id=518233498#step:5:253 is just a binutils bug according to https://bbs.archlinux.org/viewtopic.php?id=242682 and other sites... * Revert linting/formatting steps, those are done (and needed) anyway for the PR... * Nasty hack with ld, tell musl that its own includes are available via CPPFLAGS and hopefully bypass clang/llvm errors? * LDFLAGS and CPPFLAGS need to be passed to hts-sys, plus need to be surrounded by quotes since there are spaces in the variables * Be a bit more careful with env variables when clippy is running * A bit more classy to pass clippy? * Temporarily try to bypass formatting/linting again, not helping focus on musl * Passing clang_arg() to bindgen to compile the wrapper with -I/usr/local/musl/x86_64-linux-musl/include does not seem to help :/ * Hardcoded include path works for both musl and GNU bindgen building, removing .cargo/config, not necessary * Re-enabling formatting and linting Github action phases after green build on MUSL testing :) * hfile_s3.o is only included on libhts.a when ./configure is run previously according to James and Rob from the htslib team * Make sure we are in htslib directory, otherwise configure.ac and .in will not be found * Fix up redundant imports according to clippy, switch to alpine linux container, way more lightweight and less messy build process, also includes MUSL as base compiler * Explicitly cross-compile passing the right ./configure --host flag, to be generalized later * LDFLAGS was being pointed to CPPFLAGS, copy&paste hell. Also make cargo clippy happy * More clippy * Puzzled as of why this clippy error was not detected locally... also htslib compiles fine locally as well, no cross, just cargo * CPPFLAGS as an argument to make is causing trouble on Linux but not on OSX... * This include does not affect any build and it is crucial for MUSL ones * Back to non-alpine container, not sure why cargo is not being found * Disabling --no-default-features building for now, clang-sys not needed, suppress cppflags warnings * Following @pmarks advice on libz-sys, bump up both libc anb libz-sys while at it * Enable no-default-features back * Make clippy happy with libz * libz-sys should not be optional, thanks @pmarks * Cleaning up flags, trying to have libz as static to avoid //usr/lib64/libz.so.1: error adding symbols: DSO missing from command line, linking errors, thansk @wezm for the tips * Figured out next steps: need to incorporate static compilation features to both bzip2-sys and lzma-sys, otherwise MUSL will have DSO linker issues. Will implement alexcrichton/xz2-rs#62 and trifectatechfoundation/bzip2-rs#56 next * Add missing static flag to lzma-sys, getting rid of the need to export the env variable, closing into the core of the issue: rust-htslib's build.rs needs a feature-flag revamp as well... in addition to env variables, it should be able to use cfg Cargo features as well... and not sed-modify htslib's makefile if it's a static build * Remove openssl-sys crate in favor of curl-sys, which already includes a vendored openssl-sys (hopefully more static-friendly) * Temporarily comment out sed modifications against htslib's makefile, focusing on static compilation * Semver + suffix on curl is not needed * make -B, that flag was the current culprit on static compilation, bizarre since it does not seem to be related at all to the libssl errors: https://www.gnu.org/software/make/manual/html_node/Options-Summary.html * Make extra sure that -fPIC flag is passed during all compile phases * Passing println(cargo:rustc-link-lib=static=hts), works, finally :) * Cleanup cflgags_patterns mess * Works locally, migrating to docker/cross builds, MUSL has regressions after build.rs changes, regular GNU toolchain works though * Cleanup build.rs * Add clux/muslrust Dockerfile example, as referred on https://users.rust-lang.org/t/musl-cross-compilation-how-to-reuse-curl-sys-libz-sys-bzip2-sys-lzma-sys-and-other-dependency-includes-under-target/42817/4 * Revert .github workflow minor changes because 'remote rejected] musl_support -> musl_support (refusing to allow an OAuth App to create or update workflow without scope)' * Adding rust workflow from master... * Remove GitHub actions workflows for now since they are blocking my progress * Thanks @wezm for feedback on MUSL dockerfiles and https://github.com/fornwall/rust-static-builder pointer * Finally compiles MUSL, passing CFLAGS=-I/usr/local/musl/include and using build.env cross to passthrough that var * Reintroducing the GitHub actions workflow... will I be able to push it now? * Nope, pushes are still blocked by '[remote rejected] musl_support -> musl_support (refusing to allow an OAuth App to create or update workflow...' * Cleanup and update README with the required cross commands. Also cleanup the different Dockerfiles tested but no longer needed. build.rs now runs make clean before reconfiguring htslib for better reproducible builds * Add back github workflow, now that I am a project collab/admin * Make sure MUSL builds find zlib.h et al * Sans quotes and let others concatenate CFLAGS * Remove clux rust container references and stray -j40 make flag Co-authored-by: Roman Valls Guimera <brainstorm@nopcode.org>
1 parent 3a2ff4e commit 8107160

File tree

15 files changed

+233
-61
lines changed

15 files changed

+233
-61
lines changed

‎.cargo/config

Lines changed: 1 addition & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,2 @@
1-
[target.x86_64-apple-darwin]
2-
rustflags = [
3-
"-C", "link-arg=-undefined",
4-
"-C", "link-arg=dynamic_lookup",
5-
]
6-
71
[target.x86_64-unknown-linux-musl]
8-
rustflags = [
9-
"-C", "link-arg=-undefined",
10-
"-C", "link-arg=dynamic_lookup",
11-
]
12-
linker = "x86_64-linux-musl-gcc"
2+
linker = "x86_64-linux-musl-gcc"

‎.github/workflows/rust.yml

Lines changed: 18 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -76,16 +76,21 @@ jobs:
7676
github-token: ${{ secrets.GITHUB_TOKEN }}
7777
path-to-lcov: ./lcov.info
7878

79-
# - name: Test musl build without default features
80-
# uses: actions-rs/cargo@v1
81-
# with:
82-
# use-cross: true
83-
# command: build
84-
# args: --target x86_64-unknown-linux-musl --no-default-features
85-
#
86-
# - name: Test musl build with all features
87-
# uses: actions-rs/cargo@v1
88-
# with:
89-
# use-cross: true
90-
# command: build
91-
# args: --target x86_64-unknown-linux-musl --all-features
79+
- name: Test musl build without default features
80+
env:
81+
CFLAGS: -I/usr/local/musl/include
82+
uses: actions-rs/cargo@v1
83+
with:
84+
use-cross: true
85+
command: build
86+
args: --target x86_64-unknown-linux-musl --no-default-features
87+
88+
- name: Test musl build with all features
89+
env:
90+
CFLAGS: -I/usr/local/musl/include
91+
uses: actions-rs/cargo@v1
92+
with:
93+
use-cross: true
94+
command: build
95+
args: --target x86_64-unknown-linux-musl --all-features
96+

‎Cargo.toml

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ tag-message = "Version {{version}} of Rust-HTSlib."
1717

1818
[dependencies]
1919
libc = "0.2"
20-
itertools = "0.8"
20+
itertools = "0.9.0"
2121
newtype_derive = "0.1"
2222
custom_derive = "0.1"
2323
url = "2.1"
@@ -29,16 +29,17 @@ linear-map = "1.2"
2929
serde_base = { version = "^1", optional = true, package = "serde" }
3030
serde_bytes = { version = "0.11", optional = true }
3131
bio-types = ">=0.6"
32-
snafu = ">= 0.5.0, <= 0.6.0"
32+
snafu = "0.6.8"
3333
hts-sys = { version = "^1.10", path = "hts-sys", default-features = false }
3434

3535
[features]
3636
default = ["bzip2", "lzma", "curl"]
3737
bzip2 = ["hts-sys/bzip2"]
3838
lzma = ["hts-sys/lzma"]
3939
curl = ["hts-sys/curl"]
40-
openssl = ["hts-sys/openssl"]
40+
#openssl = ["hts-sys/openssl"]
4141
serde = ["serde_base", "serde_bytes"]
42+
static = []
4243

4344
[dev-dependencies]
4445
tempdir = "0.3"

‎Cross.toml

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,12 @@
1+
[build.env]
2+
passthrough = [
3+
"RUST_DEBUG",
4+
"RUST_BACKTRACE",
5+
"CFLAGS"
6+
]
7+
8+
19
[target.x86_64-unknown-linux-musl]
2-
image = "brainstorm/rust_musl_docker:stable-latest-libcurl"
10+
image = "brainstorm/cross-x86_64-unknown-linux-musl:latest"
311
[target.x86_64-unknown-linux-gnu]
412
image = "brainstorm/cross-x86_64-unknown-linux-gnu:libcurl-openssl"

‎README.md

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,16 +10,30 @@ This library provides HTSlib bindings and a high level Rust API for reading and
1010

1111
To clone this repository, issue
1212

13-
```
14-
git clone --recursive https://github.com/rust-bio/rust-htslib.git
13+
```shell
14+
$ git clone --recursive https://github.com/rust-bio/rust-htslib.git
1515
```
1616

1717
ensuring that the HTSlib submodule is fetched, too.
1818
If you only want to use the library, there is no need to clone the repository. Go on to the **Usage** section in this case.
1919

2020
## Requirements
2121

22-
To compile this crate you need the development headers of zlib, bzip2 and xz. For instance, in Debian systems one needs the following dependencies:
22+
To compile this crate you need docker and cross:
23+
24+
```shell
25+
$ cargo install cross
26+
$ cross build # will build with GNU toolchain
27+
```
28+
29+
If you want to run rust-htslib code on AWS lambda, you'll need to statically compile it with MUSL as follows:
30+
31+
```shell
32+
$ export CFLAGS="-I/usr/local/musl/include"
33+
$ cross build --target x86_64-unknown-linux-musl # will build with MUSL toolchain
34+
```
35+
36+
Alternatively, you can also install it locally by installing the development headers of zlib, bzip2 and xz. For instance, in Debian systems one needs the following dependencies:
2337

2438
```shell
2539
$ sudo apt-get install zlib1g-dev libbz2-dev liblzma-dev clang pkg-config

‎docker/Dockerfile.gnu

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,9 @@
11
FROM rustembedded/cross:x86_64-unknown-linux-gnu
22

3-
ENV LIBCLANG_PATH /usr/lib/x86_64-linux-gnu
3+
#ENV LIBCLANG_PATH /usr/lib/x86_64-linux-musl
4+
ENV LIBCLANG_PATH /usr/lib/llvm-10/lib
45
ENV LLVM_CONFIG_PATH /usr/bin
5-
RUN apt-get update && \
6-
apt-get install -y libcurl4-openssl-dev zlib1g-dev libbz2-dev liblzma-dev clang-8 && \
7-
ln -sf /usr/bin/llvm-config-8 /usr/bin/llvm-config && \
8-
ln -sf /usr/lib/x86_64-linux-gnu/libclang-8.so.1 /usr/lib/x86_64-linux-gnu/libclang.so.1
6+
RUN apt-get update
7+
RUN apt-get install -y wget gnupg lsb-release software-properties-common apt-transport-https ca-certificates # Otherwise LLVM bump below fails
8+
RUN bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
9+
RUN apt-get install -y libssl-dev libcurl4-openssl-dev zlib1g-dev libbz2-dev liblzma-dev # htslib deps

‎docker/Dockerfile.musl

Lines changed: 117 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,121 @@
11
FROM rustembedded/cross:x86_64-unknown-linux-musl
22

3+
ENV MUSL_CROSS_MAKE_VERSION 0.9.9
34
ENV PKG_CONFIG_ALLOW_CROSS 1
4-
ENV OPENSSL_LIB_DIR /usr/lib/x86_64-linux-gnu
5-
ENV OPENSSL_INCLUDE_DIR /usr/include/openssl
5+
ENV LZMA_VERSION 5.2.5
6+
ENV ZLIB_VERSION 1.2.11
7+
ENV OPENSSL_VERSION 1_1_1g
8+
ENV CURL_VERSION 7.70.0
9+
10+
ENV LIBCLANG_PATH /usr/lib/llvm-10/lib
11+
ENV LLVM_CONFIG_PATH /usr/bin
12+
13+
14+
# The default includes and packages from the Ubuntu distro that cross uses will generate all sorts of linux-headers related include errors, see:
15+
# https://github.com/rust-bio/rust-htslib/pull/184#commitcomment-37496651
16+
# Those are the packages installed, hopefully someone will find a good way to use the distro ones instead of compiling everything under /usr/local :/
17+
# apt-get install -y libssl-dev libcurl4-openssl-dev zlib1g-dev libbz2-dev liblzma-dev musl musl-dev musl-tools linux-libc-dev linux-headers-4.15.0-20-generic
18+
19+
# Install basics to locally compile htslib dependencies
620
RUN apt-get update && \
7-
apt-get install -y libssl-dev libcurl4-openssl-dev zlib1g-dev libbz2-dev liblzma-dev musl musl-dev musl-tools linux-libc-dev linux-headers-4.15.0-20-generic
21+
apt-get install -y build-essential autoconf automake autotools-dev git wget
22+
23+
# Otherwise LLVM bump below fails
24+
RUN apt-get install -y wget gnupg lsb-release software-properties-common apt-transport-https ca-certificates
25+
26+
# Autodetect and fetch latest LLVM repos for the current distro, avoids LLVM warnings and other issues, might generate slower builds for now though, see:
27+
# https://www.phoronix.com/scan.php?page=news_item&px=Rust-Hurt-On-LLVM-10
28+
RUN bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
29+
#RUN apt-get install -y libssl-dev libcurl4-openssl-dev zlib1g-dev libbz2-dev liblzma-dev musl musl-dev musl-tools # htslib deps
30+
31+
# Remove pre-installed musl, to avoid cross-musl-make interference
32+
RUN apt-get remove -y musl
33+
34+
# For now we'll have to go nuts and not only build musl-1.2.0 from scratch but all the other libs too...
35+
# Updated musl-cross toolchain that does not fail on OpenSSL: https://github.com/openssl/openssl/issues/7207
36+
WORKDIR /root
37+
RUN wget https://github.com/richfelker/musl-cross-make/archive/v$MUSL_CROSS_MAKE_VERSION.tar.gz \
38+
&& tar xvfz v$MUSL_CROSS_MAKE_VERSION.tar.gz
39+
WORKDIR /root/musl-cross-make-$MUSL_CROSS_MAKE_VERSION
40+
COPY config-musl-cross-make.mak config.mak
41+
RUN make -j40 install
42+
43+
# Now we assume we have a properly configured musl-cross...
44+
ENV PATH "/usr/local/musl/bin:$PATH"
45+
ENV CFLAGS "-fPIC"
46+
ENV CROSS_COMPILE x86_64-linux-musl-
47+
ENV CC ${CROSS_COMPILE}cc
48+
ENV AR ${CROSS_COMPILE}ar
49+
ENV RANLIB ${CROSS_COMPILE}ranlib
50+
ENV CXX ${CROSS_COMPILE}g++
51+
ENV CPPFLAGS "-I/usr/local/musl/include -I/usr/local/include"
52+
ENV LDFLAGS "-L/usr/local/musl/lib -L/usr/local/lib"
53+
54+
# .. and carry on with the htslib deps
55+
RUN wget https://www.zlib.net/zlib-$ZLIB_VERSION.tar.gz \
56+
&& tar xvfz zlib-$ZLIB_VERSION.tar.gz \
57+
&& cd zlib-$ZLIB_VERSION \
58+
&& ./configure --static --prefix=/usr/local/musl \
59+
&& make -j40 install
60+
WORKDIR /root
61+
RUN git clone git://sourceware.org/git/bzip2 \
62+
&& cd bzip2 \
63+
#&& make -j40 CC=$CC AR=$AR RANLIB=$RANLIB CFLAGS=$CFLAGS bzip2 \
64+
&& make PREFIX=/usr/local/musl -j40 bzip2 \
65+
&& make -j40 install
66+
WORKDIR /root
67+
RUN wget https://tukaani.org/xz/xz-$LZMA_VERSION.tar.bz2 \
68+
&& tar xvfj xz-$LZMA_VERSION.tar.bz2 \
69+
&& cd xz-$LZMA_VERSION \
70+
&& ./configure --prefix=/usr/local/musl --enable-static --disable-shared --host x86_64-unknown-linux-musl \
71+
&& make -j40 install
72+
WORKDIR /root
73+
74+
# A few gems from: https://wiki.openssl.org/index.php/Compilation_and_Installation
75+
# "OpenSSL has been around a long time, and it carries around a lot of cruft"
76+
# "SSLv2 is completely broken, and you should disable it during configuration"
77+
# "You should specify both --prefix and --openssldir to ensure make install works as expected."
78+
#
79+
# And also this:
80+
# https://github.com/openssl/openssl/issues/11362
81+
#
82+
# And having to redefine AR,CC & RANLIB because otherwise, this:
83+
#
84+
# make[1]: x86_64-linux-musl-x86_64-linux-musl-ar: Command not found
85+
RUN wget https://github.com/openssl/openssl/archive/OpenSSL_$OPENSSL_VERSION.tar.gz \
86+
&& tar xvfz OpenSSL_$OPENSSL_VERSION.tar.gz && cd openssl-OpenSSL_$OPENSSL_VERSION \
87+
&& ./Configure --prefix=/usr/local/musl --openssldir=/usr/local/musl \
88+
--with-zlib-lib=/usr/local/lib/zlib-$ZLIB_VERSION linux-x86_64 \
89+
no-shared no-dso no-gost no-engine no-ssl2 no-srp no-srtp no-tests zlib \
90+
no-weak-ssl-ciphers \
91+
&& make AR=x86_64-linux-musl-ar \
92+
CC=x86_64-linux-musl-cc \
93+
-j40 \
94+
&& make RANLIB=x86_64-linux-musl-ranlib -j40 install
95+
96+
WORKDIR /root
97+
RUN wget https://curl.haxx.se/download/curl-$CURL_VERSION.tar.gz \
98+
&& tar xvfz curl-$CURL_VERSION.tar.gz && cd curl-$CURL_VERSION \
99+
&& ./configure --prefix=/usr/local/musl --host x86_64-linux-musl \
100+
--with-ssl=/usr/local/musl --with-zlib=/usr/local/musl --enable-static \
101+
--disable-dict --disable-file --disable-ftp --disable-gopher --disable-imap \
102+
--disable-pop3 --disable-rtsp --disable-smb --disable-smtp --disable-telnet \
103+
--disable-tftp --disable-ntlm --disable-ldap \
104+
&& make -j40 install
105+
106+
# To cater Rust's openssl-sys needs...
107+
#ENV OPENSSL_DIR /usr/local/openssl
108+
ENV OPENSSL_DIR /usr/local/musl
109+
110+
# Hack to force ld stick to musl on the hts-sys/build.rs side, otherwise:
111+
# = note: /usr/bin/ld: /target/debug/deps/liblibloading-689161fea10b6234.rlib(global_static.o): unable to initialize decompress status for section .debug_info
112+
RUN rm /usr/bin/ld && ln -sf /usr/local/musl/bin/x86_64-linux-musl-ld /usr/bin/ld
113+
114+
# Prepare rustup and toolchain locally for easy manual intervention in this container
115+
#RUN wget https://static.rust-lang.org/rustup/dist/x86_64-unknown-linux-gnu/rustup-init \
116+
# && chmod +x rustup-init \
117+
# && ./rustup-init -y \
118+
# && . $HOME/.cargo/env \
119+
# && rustup target add x86_64-unknown-linux-musl
120+
121+
CMD ["bash"]

‎docker/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ $ docker build -t brainstorm/cross-x86_64-unknown-linux-musl:libcurl-openssl . -
1010
$ docker build -t brainstorm/cross-x86_64-unknown-linux-gnu:libcurl-openssl . -f Dockerfile.gnu
1111
```
1212

13-
Then to build and test rust-htslib with the above containers, proceed as you would with `cargo`, using `cross` instead, i.e:
13+
Then, to build and test rust-htslib with the above containers, proceed as you would with `cargo`, using `cross` instead, i.e:
1414

1515
```shell
1616
$ cross build --target x86_64-unknown-linux-musl

‎docker/config-musl-cross-make.mak

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
TARGET = x86_64-linux-musl
2+
OUTPUT = /usr/local/musl
3+
GCC_VER = 9.2.0
4+
BINUTILS_VER = 2.33.1

‎hts-sys/Cargo.toml

Lines changed: 10 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -17,22 +17,22 @@ pre-release-commit-message = "release version {{version}}"
1717
tag-message = "Version {{version}} of Rust-HTSlib."
1818

1919
[dependencies]
20-
libc = "0.2"
21-
libz-sys = "1.0"
22-
bzip2-sys = { version = "0.1", optional = true }
23-
lzma-sys = { version = "0.1", optional = true }
24-
curl-sys = { version = "0.4.26", optional = true }
25-
openssl-sys = { version = "0.9.54", optional = true }
20+
libz-sys = { version = "1.0.25", features = ["static"] }
21+
# https://github.com/alexcrichton/bzip2-rs/issues/56
22+
bzip2-sys = { version = "0.1.8", optional = true }
23+
lzma-sys = { version = "0.1.16", optional = true, features = ["static"] }
24+
curl-sys = { version = "0.4.31", optional = true, features = ["static-curl", "static-ssl"] }
2625

2726
[features]
28-
default = ["bzip2", "lzma", "curl"]
27+
default = ["bzip2", "lzma"]
2928
bzip2 = ["bzip2-sys"]
3029
lzma = ["lzma-sys"]
31-
openssl = ["openssl-sys"]
30+
#openssl = ["openssl-sys"]
3231
curl = ["curl-sys"]
32+
static = []
3333

3434
[build-dependencies]
3535
fs-utils = "1.1"
36-
bindgen = { version = "0.53.1", default-features = false, features = ["runtime"] }
36+
bindgen = { version = "0.53.2", default-features = false, features = ["runtime"] }
3737
cc = "1.0"
38-
glob = "0.3.0"
38+
glob = "0.3.0"

0 commit comments

Comments
 (0)