Dakota segfaults when running PCE with large numbers of responses.

[bitternp]

Hello. I recently tried to run a PCE for a problem with a large number of responses (two field variables with 90,000 elements each). Dakota segfaults when processing the results. It works fine when I reduce the size of the field variables to 900 elements. **Are there any compile-time limitations on problem sizes that can be run with Dakota? **

I am not running out of memory and my ulimit is set to unlimited. I am running Dakota 6.20.0 on RHEL8.

Below is a minimal example that reproduces this error for me. The driver script 'driver.sh' just fills the results.dat file with random numbers.

Any thoughts or suggestions would be much appreciated!

environment
  tabular_data
    tabular_data_file = 'output_data.dat'
  results_output
    hdf5
    results_output_file 'hdf5_output_data'

method
  polynomial_chaos
    sample_type lhs
    seed = 1
    variance_based_decomp
    expansion_order 2
    expansion_samples 4

model
  single

variables
  uniform_uncertain = 6
    initial_point     2   2   2   2   1   1
    lower_bounds      1   1   1   1   2   2
    upper_bounds      3   3   3   3   3   3
    descriptors       'A' "B" "C" "D" "E" "F"

interface
  analysis_drivers = 'driver.sh'
  fork
    parameters_file = 'params.dat'
    results_file = 'results.dat'
    work_directory named 'work'
    link_files 'template/*'

responses
  descriptors = "s1" "s2" "s3" "f1" "f2" 
  response_functions = 5
    scalar_responses = 3
    field_responses  = 2 
      lengths = 90000  90000  

  no_gradients
  no_hessians

Output:

 Expectations of 28 chaos coefficients using 4 observations.
 Expectations of 28 chaos coefficients using 4 observations.
 Expectations of 28 chaos coefficients using 4 observations.
 Expectations of 28 chaos coefficients using 4 observations.

 <<<<< global_projection_orthogonal_polynomial approximation builds completed."
Segmentation fault      (core dumped) dakota -i input.in"

bug_report.tar.gz

[bitternp]

Update: I found the bug. The 32 bit integer size is exceeded in the covariance matrix. The covariance matrix has the following type (dakota_data_types.hpp):

typedef Teuchos::SerialSymDenseMatrix<int, Real> RealSymMatrix;

The Teuchos matrix uses a size_t during allocation, so that works fine. But during access of the matrix, it uses ints, which causes an integer overflow. As a result, Dakota cannot compute a full covariance matrix for more than sqrt(2**32/2) = 46340 responses. Setting the covariance matrix to diagonal only fixes this issue.

I would recommend fixing this bug with one of the following options:

1. Change the template above to use 64 bit integers (note that this requires type changes for other variables too)
2. When running with full covariance turned on, add a check that limits the number of responses to 46340

Note that most users probably don't want the full covariance matrix with that many responses. But it is turned on by default, causing a segfault on large problems.