Upgrading to HDF5 1.14

[rcannood]

Hi @grimbough !

Is there a roadmap for HDF5 1.14 support in rhdf5? Would it make sense to first upgrade to HDF5 1.12 and then to HDF5 1.14?

Kind regards,
Robrecht

[grimbough]

Hi @rcannood

Thanks for the question. Unfortunately, the short answer is "no, there's not a roadmap or timeline".

This is mostly because there's never been much imperative to do. I haven't had many requests from users to update in order to user some of the newer features in HDF5, or to read files that have been created elsewhere and are inaccessible to users of my packages. Do you have a use case for transitioning to 1.14? If there's a concrete need then I'm more likely to look into it.

Regarding an upgrade path, IIRC I tried several years back to port to 1.12.0. However a combination of API changes that broke {rhdf5} and difficulty compiling the pre-built Windows library using the R toolchain meant I didn't progress to releasing this. This was before 1.14 was released.

I don't think there would be much more pain to jump straight to 1.14 if we wanted to upgrade. There are breaking API changes with any major HDF5 release, so probably better to just tackle them all at once. The real issue is finding the time to do this. It's not a minor undertaking, and maintaining these packages isn't really my primary job anymore, so I personally only have limited capacity.

I also sometimes get push back from people who maintain more unusual build systems or repositories, who would much rather I relied on the libhdf5 distributed by their package managers than adding complexity by building my own version in this package. Given that, I don't want to do a half-assed job of any upgrade.

[mr-c]

Hello, I help package rhdf5 for Debian. By policy we have to use the system copy of libhdf5 and recently we upgraded to libhdf5 version 1.14.

Unfortunately this causes a build error:

gcc -I"/usr/share/R/include" -DNDEBUG  -I'/usr/lib/R/site-library/Rhdf5lib/include'     -fpic  -g -O2 -ffile-prefix-map=/build/reproducible-path/r-base-4.4.2=. -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -Wdate-time -D_FORTIFY_SOURCE=2  -c H5constants.c -o H5constants.o
H5constants.c: In function ‘_H5constants’:
H5constants.c:156:36: error: ‘H5I_REFERENCE’ undeclared (first use in this function); did you mean ‘H5T_REFERENCE’?
  156 |                                    H5I_REFERENCE, H5I_VFL, H5I_GENPROP_CLS, H5I_GENPROP_LST,
      |                                    ^~~~~~~~~~~~~
      |                                    H5T_REFERENCE
H5constants.c:156:36: note: each undeclared identifier is reported only once for each function it appears in

Any patch to fix compatibility of rhdf5 with libhdf5 1.14 would be very welcome!

[mr-c]

Pardon me, another Debian contributor has a patch: https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1087826;filename=r-bioc-rhdf5.debdiff;msg=5 but it is really a hack to include a private header file into the build.

[mr-c]

Revised patch for hdf5 1.14 is at https://salsa.debian.org/r-pkg-team/r-bioc-rhdf5/-/blob/debian/experimental/debian/patches/hdf5-1.14.patch?ref_type=heads (but it does not address the usage of the private headers)

[LTLA]

FWIW now that biocmake is available, it may be possible to eliminate a lot of the manual labor by just using CMake to build the HDF5 static library. This works pretty well for Rigraphlib and might allow you to just drop in a tarball of HDF5 source code without any extra fiddling to get it to work with R's toolchain.

[LTLA]

I guess now I am incentivized to try my own suggestion, as the oldness of the vendored HDF5 library is causing performance problems (see tatami-inc/beachmat.hdf5#4). @grimbough if I can get it to work, would you be willing to consider a PR that uses biocmake to handle the build?

[LTLA]

FYI making pretty good progress on the CMake approach, if you're interested you can see the cmake branch at https://github.com/LTLA/Rhdf5lib. I think it compiles on mac and linux (haven't tested windows yet). rhdf5 breaks for reasons described above, but these should be pretty simple to fix.

[LTLA]

@grimbough Any opinions on the Cmake approach? We could also provide options in pkgconfig() to provide flags for using system libraries if an administrator insists.