Inconsistent file hashes when writing `data.frame`s

[lazappi]

We have noticed that files created with {rhdf5} can have inconsistent hashes even though the content is identical with writing a data.frame. Here is a minimal example:

library(rhdf5)

h5_path <- tempfile(fileext = ".h5")

hashes <- purrr::map_chr(1:5, \(i) {
  h5createFile(h5_path)
  h5write(data.frame(a = 1L:5L), h5_path, "df")
  hash <- rlang::hash_file(h5_path)
  h5closeAll()
  fs::file_delete(h5_path)
  
  Sys.sleep(1)
  
  hash
})

unique(hashes)
#> [1] "fe872321c9d8b174799c3d77d6ded667" "fcfb171578516d910ecd32051427b636"
#> [3] "8bf7d5edcffb0ca1e7bd89d1152241d5" "78069aa4a27f1322ae239d336fec60e5"
#> [5] "9a22f19c41919eeb43880d7f58139b6e"

^{Created on 2025-05-28 with reprex v2.1.1}

The Sys.sleep() isn't necessary for bigger examples but it makes this one more consistent. It suggests that maybe a timestamp is being saved in the file somewhere but I've also considered that it might be something to do with the order things are written in the file. I have tried the example in #51 and that is consistent so it seems to be something to do with data.frames but possibly it's more general than that.

Do you suggestions what it might be and/or how to reliably write a file with a consistent hash?

Also, if you have hints for how to check for timestamps in any HDF5 file that would be super helpful.

Thanks!

[PeteHaitch]

It seems to have something to do with the implicit DataFrameAsCompound = TRUE in the call to h5write(), which is the default value when it calls h5writeDataset() (the S3 method for class 'data.frame').
Setting DataFrameAsCompound = FALSE gives a consistent hash for this example.

library(rhdf5)

h5_path <- tempfile(fileext = ".h5")

hashes <- lapply(1:5, function(i) {
  h5createFile(h5_path)
  h5write(data.frame(a = 1L:5L), h5_path, "df", DataFrameAsCompound = FALSE)
  hash <- rlang::hash_file(h5_path)
  h5closeAll()
  fs::file_delete(h5_path)
  
  Sys.sleep(1)
  
  hash
})

unique(hashes)
#> [[1]]
#> [1] "a99352ce0706290a3e849d82e59c7429"

[lazappi]

That helps, thanks!

[grimbough]

My guess is that it's the same timestamp issue from #51, but the change introduced there doesn't apply when creating the compound dataset. There's a separate branch of C code for that case, and I think it will create a 'dataset creation property list' there, rather than one created in R which happens for most other instances. I'll check whether that's the case, and if we can also turn off the timestamp there.

[grimbough]

It looks like that was the issue. Here's the output after applying 64a9ddb

library(rhdf5)

packageVersion('rhdf5')
#> [1] '2.53.1'

h5_path <- tempfile(fileext = ".h5")

hashes <- purrr::map_chr(1:5, \(i) {
  h5createFile(h5_path)
  h5write(data.frame(a = 1L:5L), h5_path, "df")
  hash <- rlang::hash_file(h5_path)
  h5closeAll()
  fs::file_delete(h5_path)
  
  Sys.sleep(1)
  
  hash
})

unique(hashes)
#> [1] "507224b45cd18c59a1145e37d4466408"

This should be available in rhdf5 2.52.1 and 2.53.1 once they make it through the build system.

[lazappi]

Thanks! For my future reference do you know of a way to see/check for these timestamps in a HDF5 file? I tried all kinds of stuff but I couldn't find any way to display them.

[grimbough]

I think you have to use H5Oget_info(). I'm not sure I've ever exposed that in rhdf5, but maybe it's available in the python binding.

[grimbough]

Just for my own satisfaction, here's an initial implementation of H5Oget_info() which reports the timestamps. In this case I have to use some more complex code to actual use them since we just switched it off by default.

dat <- matrix(1:1000L, ncol = 10)
h5_path <- tempfile(fileext = ".h5")
h5createFile(h5_path)

fid <- H5Fopen(h5_path)
sid <- H5Screate_simple(c(10, 100))

dcpl <- H5Pcreate(type = 'H5P_DATASET_CREATE')
H5Pset_layout(dcpl, layout = "H5D_CHUNKED")
H5Pset_chunk(dcpl, dim = c(3, 20))

did2 <- H5Dcreate(fid, name = "/timestamp", h5space = sid,  
                  dtype_id = "H5T_NATIVE_INT", dcpl = dcpl)
H5Dwrite(did2, dat)

.Call("_H5Oget_info", did2@ID, PACKAGE = 'rhdf5')

h5closeAll()

$fileno
[1] 10

$atime
[1] "1970-01-01 01:00:00 CET"

$mtime
[1] "1970-01-01 01:00:00 CET"

$ctime
[1] "2025-06-03 17:00:45 CEST"

$btime
[1] "1970-01-01 01:00:00 CET"

$num_attrs
[1] 0

$type
[1] "DATASET"

This seems to set the 'Change time', but not the 'Access time', 'Modification Time' or 'Birth Time'. It's not clear to me what will set those. Under the new default settings all of them will just report the Unix Epoch time.

I'll merge this function tomorrow once it's properly implemented.

[grimbough]

The H5Oget_info() function is now in {rhdf5} v 2.53.2 and there's also an h5readTimestamps() function too:

library(rhdf5)
example_file <- system.file("testfiles", "h5ex_t_array.h5", package="rhdf5")
 
## read timestamps on a dataset
h5readTimestamps(example_file, name = '/DS1')
#> $access_time
#> [1] "1970-01-01 UTC"
#> 
#> $modification_time
#> [1] "1970-01-01 UTC"
#> 
#> $change_time
#> [1] "2018-01-11 18:31:45 UTC"
#> 
#> $birth_time
#> [1] "1970-01-01 UTC"

[lazappi]

Thanks for all the help!