Skip to content

Add support for BLAS and LAPACK dependencies (continued) #14773

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add support for BLAS and LAPACK dependencies (continued) #14773

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion ci/ciimage/arch/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ pkgs=(
doxygen vulkan-headers vulkan-icd-loader vulkan-validation-layers openssh mercurial gtk-sharp-2 qt5-tools
libwmf cmake netcdf-fortran openmpi nasm gnustep-base gettext
python-lxml hotdoc rust-bindgen qt6-base qt6-tools qt6-declarative wayland wayland-protocols
intel-oneapi-mkl
blas blas64 cblas cblas64 lapack lapack64 lapacke lapacke64
intel-oneapi-mkl intel-oneapi-openmp openblas tbb
# cuda
)

Expand Down
3 changes: 2 additions & 1 deletion ci/ciimage/fedora/image.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"env": {
"CI": "1",
"SKIP_STATIC_BOOST": "1",
"MESON_CI_JOBNAME": "linux-fedora-gcc"
"MESON_CI_JOBNAME": "linux-fedora-gcc",
"MKL_THREADING_LAYER": "GNU"
}
}
16 changes: 15 additions & 1 deletion ci/ciimage/fedora/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,29 @@ pkgs=(
doxygen vulkan-devel vulkan-validation-layers-devel openssh lksctp-tools-devel objfw mercurial gtk-sharp2-devel libpcap-devel gpgme-devel
qt5-qtbase-devel qt5-qttools-devel qt5-linguist qt5-qtbase-private-devel
qt6-qtdeclarative-devel qt6-qtbase-devel qt6-qttools-devel qt6-linguist qt6-qtbase-private-devel
libwmf-devel valgrind cmake openmpi-devel nasm gnustep-base-devel gettext-devel ncurses-devel
libwmf-devel valgrind cmake openmpi-devel nasm gnustep-base-devel gettext-devel ncurses-devel hwloc-devel
libxml2-devel libxslt-devel libyaml-devel glib2-devel json-glib-devel libgcrypt-devel wayland-devel wayland-protocols-devel
openblas-devel blas-devel lapack-devel intel-oneapi-mkl-devel
# HACK: remove npm once we switch back to hotdoc sdist
nodejs-npm
)

# Sys update
dnf -y upgrade

# Add Intel oneAPI repository for mkl
cat > /etc/yum.repos.d/oneAPI.repo <<EOF
[oneAPI]
name=Intel® oneAPI repository
baseurl=https://yum.repos.intel.com/oneapi
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
# use low priority or it will replace openmpi-devel
priority=10000
EOF

# Install deps
dnf -y install "${pkgs[@]}"
# HACK: build hotdoc from git repo since current sdist is broken on modern compilers
Expand Down
3 changes: 3 additions & 0 deletions ci/ciimage/gentoo/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ pkgs_stable=(
dev-qt/linguist-tools
sys-devel/llvm
dev-qt/qttools
sci-libs/openblas

# misc
app-admin/sudo
Expand Down Expand Up @@ -78,6 +79,7 @@ pkgs_latest=(

# ~arch only
sci-libs/scalapack
sci-libs/mkl
)
pkgs=( "${pkgs_stable[@]}" "${pkgs_latest[@]}" )

Expand Down Expand Up @@ -105,6 +107,7 @@ cat <<-EOF > /etc/portage/package.use/ci
dev-lang/rust clippy rustfmt
dev-lang/rust-bin clippy rustfmt
dev-libs/boost python
sci-libs/lapack lapacke

# Some of these settings are needed just to get the binpkg but
# aren't negative to have anyway
Expand Down
16 changes: 15 additions & 1 deletion ci/ciimage/opensuse/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ pkgs=(
boost-devel libboost_date_time-devel libboost_filesystem-devel libboost_locale-devel libboost_system-devel
libboost_test-devel libboost_log-devel libboost_regex-devel
libboost_python3-devel libboost_regex-devel
blas-devel cblas-devel lapack-devel lapacke-devel openblas-devel intel-oneapi-mkl-devel
# HACK: remove npm once we switch back to hotdoc sdist
npm
)
Expand All @@ -27,8 +28,21 @@ pkgs=(
zypper --non-interactive patch --with-update --with-optional
zypper --non-interactive update

# Add Intel oneAPI repository for mkl
cat > /etc/zypp/repos.d/oneAPI.repo <<-EOF
[oneAPI]
name=Intel® oneAPI repository
baseurl=https://yum.repos.intel.com/oneapi
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://yum.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
# use low priority or it will replace openmpi-devel
priority=10000
EOF

# Install deps
zypper install -y "${pkgs[@]}"
zypper --gpg-auto-import-keys install -y "${pkgs[@]}"
# HACK: build hotdoc from git repo since current sdist is broken on modern compilers
# change back to 'hotdoc' once it's fixed
install_python_packages git+https://github.com/hotdoc/hotdoc
Expand Down
2 changes: 2 additions & 0 deletions ci/ciimage/ubuntu-rolling/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,8 @@ pkgs=(
itstool
openjdk-11-jre
jq
libblas-dev liblapack-dev liblapacke-dev libblas64-dev liblapack64-dev liblapacke64-dev
libopenblas-dev libopenblas64-dev libmkl-dev
)

sed -i '/^Types: deb/s/deb/deb deb-src/' /etc/apt/sources.list.d/ubuntu.sources
Expand Down
146 changes: 146 additions & 0 deletions docs/markdown/Dependencies.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -327,6 +327,152 @@ to what is provided by the C runtime libraries.

`method` may be `auto`, `builtin` or `system`.

## BLAS and LAPACK

*(added 1.9.0)*

Enables compiling and linking against BLAS and LAPACK libraries. BLAS and
LAPACK are generic APIs, which can be provided by a number of different
implementations. It is possible to request either any implementation that
provides the API, or a specific implementation like OpenBLAS or MKL.
Furthermore, a preferred order may be specified, as well as the default
integer size (LP64/32-bit or ILP64/64-bit) and whether to detect the Fortran
and/or C APIs.

The common API between all BLAS and LAPACK dependencies uses the `modules`
keyword, with possible values:

- `'interface: lp64'` (default) or `'interface: ilp64'`: to select the default
integer size
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand why you did this, but it's conceptually not really correct. An "interface" is not really a module. Could we have some other way of expressing this information that is more explicit about what it does?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have a specific suggestion? From the discussion on #10921 I got the impression that it's a good enough compromise. Eli was rather opposed to adding new keyword arguments to dependency() over this, and the only clean alternative I can think of is creating a whole new module for this — and I'm not really convinced it's worth the effort. There's also the option of duplicating dependencies and having a ton of openblas-lp64, openblas-ilp64, mkl-lp64-iomp and so on — but I'm not convinced this will actually be cleaner.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking this too. CPS has the concept of "configurations" but I'm not really sure that this is a configuration either. We have a meeting tomorrow, I will bring this up with them and see what they think.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory you could do something like:

blas_dep = dependency('blas',
  modules: [...],
  configuration: {'interface': 'ilp64'})

But we'd need to properly review that to make sure it works in all (or at least most) use cases.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another big question is whether they provide different pkg-config names. That is, if there is both an openblas and openblas-ilp64 already, then using those names makes sense.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd be concerned about that because CPS configurations are not a mapping, they're an enumeration. You pick one, like debug. I kinda think that they should be a an array/list or mapping, because you could for example need an asan configuration, and really want a debug.

The first thought we had was to use "components", which again, roughly map to modules in Meson. What I'm thinking, is that really BLAS with the 32bit interface and BLAS with the 64bit interface are actually two different dependencies. Maybe it makes more sense to have depdendency('BLAS') and dependency('BLAS-64') or some similarly colored bikeshed?

- `'cblas'`: require the dependency to provide CBLAS (in addition to BLAS,
which is always present)
- `'lapack'`: require the dependency to provide LAPACK

The `get_variable` method on the returned dependency object may be used to
query the symbol suffix for the found library. Different libraries use
different symbol suffixes, and those are needed to link against the library
from C/Fortran code. Those suffixes may be fixed, like is the case for MKL and
Accelerate, or depend on the build configuration of the library, like is the
case for OpenBLAS.

Usage may look like:

```meson
dep_mkl = dependency('mkl', modules: ['interface: ilp64', 'cblas', 'lapack'])

# Or, to select one of multiple BLAS/LAPACK implementations:

blas = dependency(
['accelerate', 'mkl', 'openblas', 'blas'],
modules: ['interface: ilp64', 'cblas', 'lapack'],
)
blas_symbol_suffix = blas.get_variable('symbol_suffix')
if blas_symbol_suffix != ''
# The define here will be specific to the users code
_args_blas = ['-DBLAS_SYMBOL_SUFFIX=' + blas_symbol_suffix]
endif
blas_dep = declare_dependency(dependencies: [blas], compile_args: _args_blas)

# Adding the generic BLAS or LAPACK as a fallback, this may improve portability
blas_dep = dependency(['accelerate', 'mkl', 'openblas', 'blas')
lapack_dep = dependency(['accelerate', 'mkl', 'openblas', 'lapack')
```

Note that it is not necessary to specify both `'lapack'` and `'blas'` for the
same build target, because LAPACK itself depends on BLAS.


### Specific BLAS and LAPACK implementations

#### OpenBLAS

The `version` and `interface` keywords may be passed to request the use of a
specific version and interface, correspondingly:

```meson
openblas_dep = dependency('openblas',
version : '>=0.3.21',
modules: [
'interface: ilp64', # can be lp64 or ilp64
'cblas',
'lapack',
]
)
```

If OpenBLAS is installed in a nonstandard location *with* pkg-config files,
you can set `PKG_CONFIG_PATH`. Alternatively, you can specify
`openblas_includedir` and `openblas_librarydir` in your native or cross machine
file, this works also if OpenBLAS is installed *without* pkg-config files:

```ini
[properties]
openblas_includedir = '/path/to/include_dir' # should contain openblas_config.h
openblas_librarydir = '/path/to/library_dir'
```

OpenBLAS build configurations may include different symbol suffixes for ILP64 builds,
differences in library and pkg-config file naming conventions, and with or without
CBLAS or LAPACK. Meson will autodetect common symbol suffix conventions and validate
that, if `'cblas'` or `'lapack'` modules are requested, the found library actually
was built with support for those APIs.

OpenBLAS can be built with either pthreads or OpenMP threading. Information on
this is not available through Meson.

#### Intel MKL

MKL is built in a fixed build configuration per release by Intel, and always
includes both LP64 and ILP64, and CBLAS and LAPACK. MKL provides a number of
options for the threading layer - these can be selected through an MKL-specific
`'threading'` module:

```meson
mkl_dep = dependency('mkl',
version: '>=2023.2.0',
modules: [
interface: 'lp64', # options are 'lp64' or 'ilp64'
threading: 'seq', # options are 'seq', 'iomp', 'gomp', 'tbb'
]
)
```

In addition to all combinations of LP64/ILP64, threading, and dynamic/static
libraries, MLK provides a "Single Dynamic Library" (SDL). To select it, use the
`'sdl'` module. The SDL defaults to the LP64 interface and Intel OpenMP
(`'iomp'`) threading, with switching other interface and threading options
being accessible at runtime. When using the `'sdl'` module, either leave out
the interface and threading modules or use values that match the SDL defaults -
anything else will raise an error.

### Accelerate

Apple's Accelerate framework provides BLAS and LAPACK. It received a major update
in macOS 13.3, with LAPACK updated from 3.2 to 3.9, ILP64 support added, and
many longstanding bugs fixed. The `accelerate` dependency therefore only supports
this new version, and won't return a result on macOS <13.3 (note: the
corresponding 13.3 XCode SDK must also be installed) or if
`MACOS_DEPLOYMENT_TARGET` is set to a version lower than 13.3.

```meson
accelerate_dep = dependency('accelerate', modules: [interface: 'lp64'])
```

Note that the default, older Accelerate version can still be detected through
Meson's `appleframeworks` detection method:

```meson
dependency('Accelerate')
# Or:
dependency('appleframeworks', modules : 'Accelerate')
```
Those methods do not support any of the BLAS/LAPACK-specific modules.

### Other BLAS and LAPACK libraries

Specific support for other BLAS/LAPACK libraries is not (yet) included. They
can still be detected via the regular pkg-config or CMake support.

## Blocks

Enable support for Clang's blocks extension.
Expand Down
8 changes: 8 additions & 0 deletions mesonbuild/dependencies/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
BuiltinDependency, SystemDependency, get_leaf_external_dependencies)
from .detect import find_external_dependency, get_dep_identifier, packages, _packages_accept_language


__all__ = [
'Dependency',
'InternalDependency',
Expand Down Expand Up @@ -227,6 +228,13 @@ def __init__(self, name: str, environment: 'Environment', kwargs: T.Dict[str, T.
'libssl': 'misc',
'objfw': 'misc',

# From blas_lapack:
'accelerate': 'blas_lapack',
'blas': 'blas_lapack',
'lapack': 'blas_lapack',
'mkl': 'blas_lapack',
'openblas': 'blas_lapack',

# From platform:
'appleframeworks': 'platform',

Expand Down
Loading
Loading