1912/5 cartesian product

[MarcelKoch]

This PR allows defining the benchmark JSON input via cartesian products. The original behavior of creating the cartesian product between the input json and the CLI flags such as format (spmv) or operation (blas) is now emulated. Here is an example for defining the input as a cartesian product:

{
  "operator": {
    "stencil": {
      "size": 100,
      "name": "7pt"
    }
  },
  "format": ["coo", "csr", "ell"],
  "reorder": ["rcm", "amd"]
}

The cartesian product is taken between all top-level values of the input JSON. The JSON has to be a dict for this to work. Any value that is not an array will be transformed into an array with a single item.

Note that this allows for more combinations than before, since an arbitrary number of arrays can be used in the cartesian product.

[yhmtsai]

should we wrap it into "cartesian" to make it more clear and for future features?
like

{
  "operator": {
    "stencil": {
      "size": 100,
      "name": "7pt"
    }
  },
  "cartesian": {
      "format": ["coo", "csr", "ell"],
      "reorder": ["rcm", "amd"]
  }
}

[MarcelKoch]

I think your suggestion should be

{
  "cartesian": {      
      "operator": {
        "stencil": {
          "size": 100,
          "name": "7pt"
        }
      },
      "format": ["coo", "csr", "ell"],
      "reorder": ["rcm", "amd"]
  }
}

since the operator is also part of the cartesian product. TBH, I find this a bit redundant, since the cartesian keyword doesn't really add anything.

[yhmtsai]

I was thinking whether it will give easier path when we would like to introduce specific combination.

[MarcelKoch]

I was thinking whether it will give easier path when we would like to introduce specific combination.

Could you be more specific about what you mean by specific combination? If you mean that we should allow cartesian products with keys that are not on the top-level (e.g. the format key in operator), then I don't think we should provide that. This would be quite complex, and it's probably better done by some script the user writes.

[yhmtsai]

no, whether we can provide like
"reorder-format": ["rcm-coo", "amd-csr"] not [rcm, amd] x [coo, csr, ell]

[MarcelKoch]

I think this is similar to what I mentioned above (although I meant the size keyword within operator instead of format). What you describe should be done by the user, either by just defining the list of json inputs, or by generating these combination through their own script.

[MarcelKoch]

Side note: what you mention was also not possible before, so this isn't removing any existing functionality.

[yhmtsai]

stencil_input is equal to the oirignal one, but stencil_input_all will generate 5*5 kinds not 13 kinds, right?

[yhmtsai]

I think it will be the good use case to think whether we should support this kind or just say users need to expand all by them self.
Something like the following might be doable?

"keyword*": [
{
"from": "csr"
"to": "coo"
},
{
"from": "coo"
"to": "coo"
}
]

when match keyword, we will not add the "keyword*" to the resulting list but just put the underlying ones.

[MarcelKoch]

Regarding your first question, yes it will generate 5*5 operations. However, any operation that is not supported (e.g. ell -> hybrid) is just skipped.
Regarding your second point, sure it is probably possible, but I think it might be a bit too confusing to use.

[yhmtsai]

I see. could we still make this test as previous 13 kinds? it also check whether we still have the same ability as original.

[MarcelKoch]

The test uses the same 13 conversions as before. You can check it in the conversion.all.stdout.

[MarcelKoch]

It should be all. The diff with develop shows no missing conversion.

[yhmtsai]

I mean that we only need to test exactly 13 kinds as previous one. cartesian product already be test via [csr, coo] x [csr, coo]
It only shows it is not supported with the new combination.
Also, keeping it as the original one can check whether we still support list input not cartesian product input.

[MarcelKoch]

Could you perhaps rephrase your overall point in this comment chain?

[yhmtsai]

stencil_input = generate_input(["coo", "csr"]); <- test cartesian product and all function are available
stencil_input_all = generate_input(["coo", "csr", "ell", "sellp", "hybrid"]) <- test more combination than the original test case but the additional functionality are not supported.
I think stencil_input_all can stay as the same as the original one in develop branch.
i.e.

[
  {// case 1},
  {// case 2},
  {// case 3},
  {// functionality we actually support}
]

It will also be served as original array method check.

[MarcelKoch]

I added now an input file with the full list of supported conversions.

[yhmtsai]

Suggested change

	* values of the input dict. Any value that isn't a list will be converted into
	* a list of a single item.
	* values of the input dict. Any value that isn't a list will be considered as
	* a list of a single item during expanding.

I was thinking it is produce "A": ["B"] in the resulting file from "A": "B", but it is considered as "A": ["B"] during processing and lead "A": "B" in all, right?

[MarcelKoch]

Consider this input:

{
  "a": "b",
  "c": ["d"]
}

it will produce

[
  {
    "a": "b",
    "c": "d"
  }
]

Does that answer your question?

[yhmtsai]

here need to handle keyword if we allow the manual set