Running generate_pseudo_pop() in parallel?

[macork]

I was hoping to find a way to run the generate_pseudo_pop() function from CausalGPS in parallel using functions like mclapply. I am currently unable to do so, does anyone have any suggestions on how to parallelize so that I can run the package over many different values for delta (one of the parameters that you can tune)?

[Naeemkh]

@macork, CausalGPS is reasonably optimized for multicore systems (e.g., your laptop, or one server node on HPC). So you cannot get more out of it by adding a new parallelization approach. You can simply use lapply if you need to run the function on a list of parameters. The only reason to run multiple generate_pseudo_pop concurrently is to have access to multiple server nodes (e.g., on HPC). So far, there was no feature request to go beyond one node. Please let me know if you entertain the idea, then we can plan on going beyond one server node.

[macork]

Hi @Naeemkh thanks for the response. The reason I would like to add a parallelized approach is that I would like to run through generate_pseudo_pop many times with different values for the delta and scale parameters, and then select the delta and scale value that results in the lowest mean absolute correlation. It sounds like what you are saying is that to run through the function even once, it will use all of the cores in my laptop or on a server node on HPC, even if I only specify one thread. I think that running multiple generate_psuedo_pop concurrently would be a useful feature for most people if they are trying to optimize the delta and scale parameters. But I just want to confirm that they way it is structure currently that isn't possible

[Naeemkh]

@macork, technically you can do that. And if you pass only one core for the n_thread variable, it will use only that one thread, not all available resources.
I would suggest using lapply for all your optimization parameters and using all available cores internally. Because of two reasons:

1. In a very large dataset, you will run out of memory. Although memory consumption is very optimized, you need at least that much memory for each data. When you export these different data to each core, it will be a burden on memory.
2. The most demanding part of CausalGPS uses OpenMP to distribute the jobs. By passing only one thread, you remove that potential from your processing.

Having said that, this is a feature that we are planning on adding, and it might fit well for an MPI application. Probably an MPI wrapper function in the NSAPHUtils package.

Please see this discussion: #127

Please see the code:

# Question: How to run generate_pseudo_pop in lapply and parLapply?

# Answer: See the following code. However, parLapply is not recommended. Because
# causal GPS internally can use all available cores internally so you might not
# gain any performance by adding another parallel layer.

# You can either Use a wrapper function to run generate_syn_data in
# parallel or use lapply directly and provide the list variable at first,
# then the function and then other variables.


delta_n <- c(1:10)*0.1
m_d <- generate_syn_data(sample_size = 200)

wrapper_func <- function(delta_n){
  pseudo_pop <- generate_pseudo_pop(m_d$Y,
                                    m_d$treat,
                                    m_d[c("cf1","cf2","cf3","cf4","cf5","cf6")],
                                    ci_appr = "matching",
                                    pred_model = "sl",
                                    sl_lib = c("m_xgboost"),
                                    params = list(xgb_nrounds=c(10,20,30),
                                                  xgb_eta=c(0.1,0.2,0.3)),
                                    nthread = 1,
                                    covar_bl_method = "absolute",
                                    covar_bl_trs = 0.1,
                                    covar_bl_trs_type = "mean",
                                    max_attempt = 1,
                                    matching_fun = "matching_l1",
                                    delta_n = delta_n,
                                    scale = 0.5)

  return(pseudo_pop)
}

# Function to collect the results
print_mean_covariate_balance <- function(delta_n, object_list){
  m_cov <- unlist(lapply(object_list,
                         function(x){x$adjusted_corr_results$mean_absolute_corr}))
  results <- data.frame(delta_n, m_cov)
  print(results)
}


# with lapply ------------------------------------------------------------------
pseudo_pop_list_1  <- lapply(delta_n, wrapper_func)

# mean covariate balance
print_mean_covariate_balance(delta_n = delta_n, object_list = pseudo_pop_list_1)


# with parLapply ---------------------------------------------------------------
cl <- parallel::makeCluster(10, type="PSOCK")
parallel::clusterExport(cl=cl,
                        varlist = c("generate_pseudo_pop",
                                    "wrapper_func",
                                    "m_d"
                        ),
                        envir=environment())

pseudo_pop_list_2  <- parallel::parLapply(cl,delta_n, wrapper_func)
parallel::stopCluster(cl)

# mean covariate balance
print_mean_covariate_balance(delta_n = delta_n, object_list = pseudo_pop_list_2)

[macork]

Hi Naeem, thanks for posting this code it is really helpful! I will give this a try and am interested in discussion 127 because I agree it would be a nice feature