Skip to content

Conversation

MarcelKoch
Copy link
Member

This PR adds JSON schema definitions for the new benchmark input.

@MarcelKoch MarcelKoch self-assigned this Aug 15, 2025
@ginkgo-bot ginkgo-bot added reg:build This is related to the build system. reg:benchmarking This is related to benchmarking. labels Aug 15, 2025
@MarcelKoch MarcelKoch changed the title 1912: Add schema 1912/1 item schema Aug 15, 2025
"type": "string",
"pattern": "^[a-z0-9]+-[a-z0-9]+"
},
"reorder": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we only have natural reorder, should we provide this option now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since the reorder key is not required, this only checks if the reorder key is provided, then it has to be natural.

@MarcelKoch MarcelKoch changed the base branch from develop to 1912/0-identity-config August 19, 2025 14:36
Comment on lines +28 to +32
"if": {
"required": [
"stride"
]
},
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you share the documentation for this?
I interpret it like if required stride then ...
but it never require stride right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe this can help: https://json-schema.org/understanding-json-schema/reference/conditionals#ifthenelse.
I think the if tries to check if anything specified here validates. So it checks if the required validates, and based on that chooses the then or else branch.

ginkgo
gflags
nlohmann_json::nlohmann_json
nlohmann_json_schema_validator
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also need to add the license from nlohmann_json_schema_validator

"stencil": {
"type": "object",
"properties": {
"name": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"name": {
"discretization": {

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO name is shorter and just as descriptive (with the right documentation). I will leave it as is.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about kind? I.e. what kind of stencil is it? type conflicts with other categories of types, but kind is not something we have right now.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's better, thanks for the suggestion!

"filename": {
"type": "string"
},
"stencil": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will also lead the format changes. Is it intentional?
previously,

"stencil": "5pt",
"local_size" ....

now

"stencil": {
  "name": "5pt",
  ...
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, because I find this more consistent. But if others prefer the old version, I can revert.

"properties": {
"format": {
"enum": [
"coo",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we also remove the format out of CLI?
some of them might be hard to understand without comment

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It will be removed from the CLI, but there will be extra documentation to make up for it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same argument as for solvers: I think this is an error that can be reasonably caught inside the benchmark code, unless we consider it unlikely that there will be other SpMV implementations we want to test temporarily. Right now, we have two places we need to modify if we add a new matrix format, this would make it three.

)
endif()

include(FetchContent)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FetchContent things should happen in third_party ideally, and this would make building Ginkgo in air-gapped systems difficult. Since this is a library we don't export, it may be reasonable to vendor it, but if we didn't, then we would need to add it to spack etc. if we ever want to make the benchmark executables installable artifacts.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. This was more of a temporary solution that I forgot about.

"stencil": {
"type": "object",
"properties": {
"name": {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about kind? I.e. what kind of stencil is it? type conflicts with other categories of types, but kind is not something we have right now.

"properties": {
"format": {
"enum": [
"coo",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same argument as for solvers: I think this is an error that can be reasonably caught inside the benchmark code, unless we consider it unlikely that there will be other SpMV implementations we want to test temporarily. Right now, we have two places we need to modify if we add a new matrix format, this would make it three.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reg:benchmarking This is related to benchmarking. reg:build This is related to the build system.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants