Read Large sas7bdat

[amcenroe]

Hi Dai ZJ / Authors,

i saw on there was functions sas_to_csv / sas_to_disk.frame on version 0.1.0 but can't find those functions at the latest version, is it deprecated or any similar functions ?

[xiaodaigh]

I should remove them. There's no plan to develop them at this stage.

[amcenroe]

ok noted, then any idea how to read large sas7bdat using disk.frame ?

[arnaud-feldmann]

tidyverse/haven#633 One could use this basic chunked function I guess, There is no multithreading through, so it could be quite slow

[xiaodaigh]

large sas7bdat using disk.frame

There are technologies for converting sAS to CSV at speed. DM me if you are interested. It's a commercial software though.

[arnaud-feldmann]

No thanks, if I use that I'm able to convert SAS files to disk frames. And I want to rely only on open-source software. As long as it's once and that it works I don't care if it's a bit slow. There's some strange warnings though I'll check what happens

[arnaud-feldmann]

Anyway, using the read_sas_chunked of the PR above

read_sas_diskframe <- function(sas_file,outdir = tempfile(fileext = ".df"),chunk_size = 10000, shardby = NULL) {
    df <- as.disk.frame(
      read_sas(sas_file,
               n_max = 0),
      outdir = outdir,
      overwrite = TRUE,
      shardby = shardby,
      compress = 50)
    
    read_sas_chunked(sas_file,
                     chunk_size = 10000,
                     callback = \(x) {
                       add_chunk(df,x)
                       invisible(NULL)
                     })
    
    df
}

That seems to work for medium sized files, and can be enough if, like me, you mainly look for a way to handle big sas files from within R without external conversions or dependencies. But haven definitely needs a chunked API