Coliny DIRect not performing as it should ?

[jacquestest]

I am trying to use coliny DIRect as a global/local search pattern. I want to ensure the parameter space is globally sampled, hence using a global balance parameter of 0,4 (to ensure the smallest size region is 0.4 times the largest :
method coliny_direct scaling convergence_tolerance = 1.e-3 max_iterations = 200 global_balance_parameter = 0.4 min_boxsize_limit = 0.01

I let the division method as default, to divide the largest edge.
I have a 3 parameter space with a two objective function summed into one score.

The results seems okish but at the same time very weird : the parameter space is not equally sampled in the 3 parameters (one is heavily overrepresented), and the size of the regions don't abide by the 0.4 criteria by far.

Here are the results :
%eval_id interface param_1 param_2 param_3 obj_fn_1 1 NO_ID 0.5 0.5 0.5 17216.53239 2 NO_ID 0.8333333333 0.5 0.5 42833.79334 3 NO_ID 0.1666666667 0.5 0.5 794.3825515 4 NO_ID 0.1666666667 0.8333333333 0.5 9528.495281 5 NO_ID 0.1666666667 0.1666666667 0.5 12925.85696 6 NO_ID 0.1666666667 0.5 0.8333333333 1339.1325 7 NO_ID 0.1666666667 0.5 0.1666666667 6353.732106 8 NO_ID 0.6111111111 0.5 0.5 26098.2024 9 NO_ID 0.3888888889 0.5 0.5 8589.463374 10 NO_ID 0.1666666667 0.6111111111 0.5 289.897838 11 NO_ID 0.1666666667 0.3888888889 0.5 3662.319323 12 NO_ID 0.3888888889 0.5 0.8333333333 26066.89112 13 NO_ID 0.3888888889 0.5 0.1666666667 1022.786798 14 NO_ID 0.9444444444 0.5 0.5 50337.2167 15 NO_ID 0.7222222222 0.5 0.5 34728.49052 16 NO_ID 0.1666666667 0.6111111111 0.6111111111 1216.443499 17 NO_ID 0.1666666667 0.6111111111 0.3888888889 291.46587 18 NO_ID 0.3888888889 0.8333333333 0.1666666667 24258.8448 19 NO_ID 0.3888888889 0.1666666667 0.1666666667 6858.777553 20 NO_ID 0.537037037 0.5 0.5 20209.39728 21 NO_ID 0.462962963 0.5 0.5 14255.44612 22 NO_ID 0.2777777778 0.6111111111 0.5 7341.327745 23 NO_ID 0.05555555556 0.6111111111 0.5 10350.34576 24 NO_ID 0.1666666667 0.5 0.6111111111 110.02567 25 NO_ID 0.1666666667 0.5 0.3888888889 2162.781091 26 NO_ID 0.1666666667 0.6111111111 0.8333333333 6225.193383 27 NO_ID 0.1666666667 0.3888888889 0.8333333333 16.6613725 28 NO_ID 0.3888888889 0.8333333333 0.5 48425.97234 29 NO_ID 0.3888888889 0.1666666667 0.5 1004.937722 30 NO_ID 0.462962963 0.5 0.8333333333 34957.98998 31 NO_ID 0.462962963 0.5 0.1666666667 2883.435146 32 NO_ID 0.6481481481 0.5 0.5 29024.33514 33 NO_ID 0.5740740741 0.5 0.5 23138.54761 34 NO_ID 0.1666666667 0.3888888889 0.9444444444 312.909614 35 NO_ID 0.1666666667 0.3888888889 0.7222222222 570.631571 36 NO_ID 0.462962963 0.8333333333 0.1666666667 32922.97012 37 NO_ID 0.462962963 0.1666666667 0.1666666667 4302.717006 38 NO_ID 0.512345679 0.5 0.5 18210.80621 39 NO_ID 0.487654321 0.5 0.5 16222.94504 40 NO_ID 0.7222222222 0.8333333333 0.5 86607.03497 41 NO_ID 0.7222222222 0.1666666667 0.5 3182.759411 42 NO_ID 0.2777777778 0.3888888889 0.8333333333 4534.153179 43 NO_ID 0.05555555556 0.3888888889 0.8333333333 12184.54755 44 NO_ID 0.7222222222 0.1666666667 0.8333333333 15668.79588 45 NO_ID 0.7222222222 0.1666666667 0.1666666667 475.543617 46 NO_ID 0.487654321 0.8333333333 0.5 62260.21017 47 NO_ID 0.487654321 0.1666666667 0.5 117.645425 48 NO_ID 0.8333333333 0.5 0.8333333333 70159.56499 49 NO_ID 0.8333333333 0.5 0.1666666667 19952.27307 50 NO_ID 0.2037037037 0.3888888889 0.8333333333 543.383489 51 NO_ID 0.1296296296 0.3888888889 0.8333333333 1139.858752 52 NO_ID 0.487654321 0.1666666667 0.8333333333 4226.228431 53 NO_ID 0.487654321 0.1666666667 0.1666666667 3623.447253 54 NO_ID 0.5 0.8333333333 0.5 63823.62357 55 NO_ID 0.5 0.1666666667 0.5 124.854844 56 NO_ID 0.549382716 0.5 0.5 21197.81597 57 NO_ID 0.524691358 0.5 0.5 19212.3477 58 NO_ID 0.9444444444 0.8333333333 0.5 102029.5152 59 NO_ID 0.9444444444 0.1666666667 0.5 9452.627453 60 NO_ID 0.1666666667 0.3888888889 0.8703703704 14.779759 61 NO_ID 0.1666666667 0.3888888889 0.7962962963 112.785606 62 NO_ID 0.5041152263 0.1666666667 0.5 132.2461005 63 NO_ID 0.4958847737 0.1666666667 0.5 119.9207885 64 NO_ID 0.512345679 0.8333333333 0.5 65354.06883 65 NO_ID 0.512345679 0.1666666667 0.5 154.1595815 66 NO_ID 0.5740740741 0.5 0.8333333333 47186.92799 67 NO_ID 0.5740740741 0.5 0.1666666667 7128.655174 68 NO_ID 0.1790123457 0.3888888889 0.8703703704 169.4630355 69 NO_ID 0.1543209877 0.3888888889 0.8703703704 40.9168585 70 NO_ID 0.4958847737 0.1666666667 0.8333333333 4560.837392 71 NO_ID 0.4958847737 0.1666666667 0.1666666667 3414.616939 72 NO_ID 0.524691358 0.8333333333 0.5 66841.91901 73 NO_ID 0.524691358 0.1666666667 0.5 204.559062 74 NO_ID 0.6234567901 0.5 0.5 27075.50122 75 NO_ID 0.5987654321 0.5 0.5 25116.81701 76 NO_ID 0.1666666667 0.4259259259 0.8703703704 280.662852 77 NO_ID 0.1666666667 0.3518518519 0.8703703704 85.4499225 78 NO_ID 0.5 0.1666666667 0.8333333333 4730.817081 79 NO_ID 0.5 0.1666666667 0.1666666667 3314.789442 80 NO_ID 0.5411522634 0.5 0.5 20547.3679 81 NO_ID 0.5329218107 0.5 0.5 19878.21548 82 NO_ID 0.6481481481 0.8333333333 0.5 80005.19816 83 NO_ID 0.6481481481 0.1666666667 0.5 1672.240916 84 NO_ID 0.1790123457 0.3888888889 0.8333333333 40.734014 85 NO_ID 0.1543209877 0.3888888889 0.8333333333 173.538813 86 NO_ID 0.5041152263 0.1666666667 0.8333333333 4903.145882 87 NO_ID 0.5041152263 0.1666666667 0.1666666667 3215.098807 88 NO_ID 0.5329218107 0.8333333333 0.5 67817.64485 89 NO_ID 0.5329218107 0.1666666667 0.5 249.596637 90 NO_ID 0.549382716 0.5 0.8333333333 44595.85907 91 NO_ID 0.549382716 0.5 0.1666666667 6065.778497 92 NO_ID 0.170781893 0.3888888889 0.8333333333 5.7304405 93 NO_ID 0.1625514403 0.3888888889 0.8333333333 47.5680735 94 NO_ID 0.512345679 0.1666666667 0.8333333333 5251.132585 95 NO_ID 0.512345679 0.1666666667 0.1666666667 3024.913991 96 NO_ID 0.537037037 0.8333333333 0.5 68294.1441 97 NO_ID 0.537037037 0.1666666667 0.5 275.0734445 98 NO_ID 0.6028806584 0.5 0.5 25443.34386 99 NO_ID 0.5946502058 0.5 0.5 24784.98587 100 NO_ID 0.170781893 0.4259259259 0.8333333333 170.580716 101 NO_ID 0.170781893 0.3518518519 0.8333333333 169.178876 102 NO_ID 0.524691358 0.1666666667 0.8333333333 5788.965344 103 NO_ID 0.524691358 0.1666666667 0.1666666667 2753.257273 104 NO_ID 0.5425240055 0.5 0.5 20654.8077 105 NO_ID 0.5397805213 0.5 0.5 20436.57273 106 NO_ID 0.6111111111 0.8333333333 0.5 76368.82421 107 NO_ID 0.6111111111 0.1666666667 0.5 1069.943739 108 NO_ID 0.170781893 0.3888888889 0.8456790123 8.713165 109 NO_ID 0.170781893 0.3888888889 0.8209876543 13.5271535 110 NO_ID 0.1680384088 0.3888888889 0.8333333333 10.8026155 111 NO_ID 0.1652949246 0.3888888889 0.8333333333 24.627505 112 NO_ID 0.5329218107 0.1666666667 0.8333333333 6138.333459 113 NO_ID 0.5329218107 0.1666666667 0.1666666667 2582.201922 114 NO_ID 0.5397805213 0.8333333333 0.5 68612.5718 115 NO_ID 0.5397805213 0.1666666667 0.5 293.1166225 116 NO_ID 0.6275720165 0.5 0.5 27407.74763 117 NO_ID 0.6193415638 0.5 0.5 26753.10951 118 NO_ID 0.170781893 0.4012345679 0.8333333333 22.724495 119 NO_ID 0.170781893 0.3765432099 0.8333333333 25.1806185 120 NO_ID 0.1684956561 0.3888888889 0.8333333333 9.2952765 121 NO_ID 0.1675811614 0.3888888889 0.8333333333 12.491827 122 NO_ID 0.5397805213 0.1666666667 0.8333333333 6447.048575 123 NO_ID 0.5397805213 0.1666666667 0.1666666667 2443.928651 124 NO_ID 0.5411522634 0.8333333333 0.5 68776.89722 125 NO_ID 0.5411522634 0.1666666667 0.5 302.6203485 126 NO_ID 0.1686480719 0.3888888889 0.8333333333 8.918929 127 NO_ID 0.1683432404 0.3888888889 0.8333333333 9.683265 128 NO_ID 0.487654321 0.1666666667 0.6111111111 477.8143135

Side questions : why is my initial point not taken into account ? It starts with 0.5 0.5 0.5 while I gave 0,63 0,126 0,156...
Maybe my .in file has issues and is not parsed correctly ? Here is the full file :
`environment

single

graphics

tabular_data
tabular_data_file = 'res_final_optim.dat'

method
coliny_direct
scaling
convergence_tolerance = 1.e-3
max_iterations = 200
global_balance_parameter = 0.4
min_boxsize_limit = 0.01

model
single

variables

continuous_design=3
initial_point 0.63 0.126 0.156
lower_bounds 0.01 -0.4 -0.1
upper_bounds 0.9 0.8 1.1
descriptors 'param_1' 'param_2' 'param_3'
scale_types = 'auto' 'auto' 'auto'

interface,
fork
asynchronous
evaluation_concurrency = 1
analysis_driver = 'python3 ../simulator_script.py'
parameters_file = 'params.in'
results_file = 'results.out'
work_directory directory_tag
copy_files = 'templatedir/*'

uncomment to leave params.in and results.out files in work_dir subdirectories

#named 'workdir' file_save  directory_save
named 'workdir'
aprepro

when using conmin_frcg (above) with analytic_gradients (below),

need to turn off the active set vector as rosenbrock_bb does not parse it.

deactivate active_set_vector

responses
objective_functions = 2
numerical_gradients
fd_step_size = 0.02
numerical_hessians`

[jacquestest]

I dug deeper and tried the simple rosenbrock problem with this config fiule :
`environment
tabular_data
tabular_data_file = 'rosen_opt_patternsearch.dat'

method
coliny_direct
scaling
max_iterations 500
max_function_evaluations 5000
local_balance_parameter 0.03
global_balance_parameter 0.1

model
single

variables
continuous_design = 2
lower_bounds = -2.0 -2.0
upper_bounds = 2.0 2.0
descriptors = 'x1' 'x2'

interface
analysis_drivers = 'rosenbrock'
direct

responses
objective_functions = 1
no_gradients
no_hessians
`

Results are very bad : neither the local nor global balance parameter do what they are supposed to do. I can't limit the min size nor balance global/local searches.

What am I missing ? could someone provide me with a config file working for the rosenbrock problem with for example a 0.01, 0.1 and 0.5 global parameters (meaning respectively the smallest region not smaller than 0.01/0.1 or 0.5 times the largest one, the last being almost homogeneously sampled and the first very focused on interest regions.

[thenectarine]

I have not used these balance parameters before, but I was curious to see how the samples are distributed.

For the rosenbrock example that was provided, the samples do appear equally distributed. See figure 1, left. When I use division=all_dimensions, the samples also appear equally distributed, see figure 1, right. The graph in figure 1 plots x1 vs. x2.

You could try creating similar plots for the 3 variable example. While these graphs are projections from 3D to 2D, I think the graphs can reveal if the samples are equally spaced.

• Graph 1 - param_1 vs param_2
• Graph 2 - param_2 vs param_3
• Graph 3 - param_1 vs param_3

Figure 1 - Left: Original input file with division set as the default. Right: divisions=all_dimensions

[jacquestest]

Thanks for answering ! As for your two examples, I have similar results using the all/major dimension keywod. However I still think it's not performing as it should.

1. The figure 1 on the left is not doing what it is supposed, in my comprehension. Not using the all_dimensions should lead to the largest edge being refined when a subregion is divided. It should create anisotropy in the regions of 'no' interest contrarily to the uniformish sampling when using all_dimensions.
   What I see on your left plot is a distribution heavily skewed towards exploring x1, in regions of poor interest. It keeps dividing the subregions "vertically" instead of dividing vertically once, then horizontally, etc etc. Furthermore, this region is almost flat and should not be explored, especially not with this "line behaviorr" (it is particularly inefficient if you try using it on steeper functions... My suspicion is that it does the opposite, and refines subregions based on the "smallest" edge instead of the major dimension one, leading to such weird behavior. I am trying to reprogram the DIRect algorithm in a simpler way to try and test this idea.

I agree it does somewhat better close to the interesting regions (1,1) (1.7,1.7) but again weird behaviors occur and it fails to find the (1,1) minimum (NCSU does it flawlessly in 250ish steps, or other staggered researchs).

3. the local_ and global_ parameters, supposedly limiting the min hypercube/subregion size, and the global one for controlling the local to global exploration. Compared to NCSU DIRect, which do not offer such controls, it seems very useful. However, neither performs as it should (or as I understand it should). Specifically, the global parameter of say 0.1 doesn't prevent subregions 10 times smaller than the largest one to be refined (neither with major/all dimension). Weirdly, a global parameter of 20 seems to give better results to focus more on a global search...

4. the local parameter performs a bit better : values of 0.05 to 0.3 seems to somewhat prevent too small subregions to be explored. It's not tied to the size of the search domain however. My latest suspicion is that the local parameter acts as the global parameter should (prevent division is size_dividedRegion < parameter * size_maxRegion).

I would very much like to have your insight on points 2 and 3, and your answer to point 1 where the "line exploration behavior" on the left plot, on the left seems to me very uncorrect compared to what is expected.

TLDR : for now I use NCSU which does what it is supposed to, even though I can't fine tune the controls. I also use local_parameter 0.3 sometimes as it seems to be ok, but I can't rely on it blindly due to the issues mentionned.

1. For coliny_DIRect : using all_dimensions and not specifying global/local or just local parameters of 0.05 to 0.3 seems okish but not a lot of control is given.
2. NCSU does not have control options but performs as expected.

I will provide the results and images of all my tests too, and in case I successfully code a simple DIRect show what I was expecting

On another note, I wish there way a way to add logs of the size of the regions, subdivisions, and criteria that made refine each region each time step : it would allow to understand and debug in case of need what is happening. Here we just go blind...

[jacquestest]

Update : I am trying to figure out what is happening in the source files.
I am probably mistaken but shouldn't the colinSolver definition file https://github.com/snl-dakota/dakota-packages/blob/d53743b96abc5588568e074cab944a867d8d281b/acro/packages/colin/src/colin/solver/ColinSolver.h#L268 include the max_boxsize_ratio (cf

dakota/src/COLINOptimizer.cpp

Line 670 in a0b2f27

const Real& min_box = probDescDB.get_real("method.min_boxsize_limit");

:
const Real& local_bal_param = probDescDB.get_real("method.coliny.local_balance_parameter"); if (local_bal_param >= 0. && colinSolver->has_property("max_boxsize_ratio")) colinSolver->property("max_boxsize_ratio") = local_bal_param; ?

Also in the dataMethod.cpp, there are those lines constantPenalty(false), globalBalanceParam(-1.), localBalanceParam(-1.), maxBoxSize(-1.), minBoxSize(-1.), that I don't understand...

So far, no clear answer as I am having trouble navigating the source code (I didn't even find where the test using colinSolver->property("max_boxsize_ratio") and colinSolver->property("min_boxsize_ratio") are made in the actual solver iterations.